A Hard Drill Makes an Easy Battle

Tuesday, November 20, 2001

I can't say enough nice things about VMWare. This program has been amazingly helpful during the last few weeks as we tried to get CityDesk to work on every known version of 32-bit Windows. I have set up dozens of virtual machines, everything from a simple DOS partition (helpful as a starting point for installing other OSs), a bunch of combinations of NT 4.0, Chinese and Hebrew Win2K (even though our program is in English and doesn't do anything fancy, we had various bugs that were revealed on these systems), assorted versions of Win 95/98/Me going all the way back to the August, 1995 release, even a small network of machines with a primary domain controller which we used for testing FogBUGZ setup.

Getting code to work on the entire universe of Windows machines is a lot of work. That's the real appeal of "write once, run anywhere" systems like Java. In theory, if you use the Java Virtual Machine, the burden is on the VM vendor to provide compatibility with all these platforms. In reality, as Java programmers learned, code is just too fragile for this to work very well. When I developed a game with Java I learned that Java's inability to guarantee exactly when threads would run (a seemingly harmless concession to the fact that CPU scheduling is basically unpredictable) actually meant that on the Macintosh, some threads got starved, um, forever, basically, until the other threads tried to do i/o in fact, which is not what I had assumed and made my game not very challenging on Macs. (This was in 1996. Don't email me with workarounds, fixes, or to say that this bug has been fixed.)

Yesterday's Bug Du Jour is an example of the kind of thing that trips people up. Michael was allocating some memory using the ancient Windows API GlobalAlloc. Later, he was calling a function GlobalSize to determine the size of that memory. On our development systems (Windows 2000) GlobalSize returns the same value as the block you allocated. Allocate 13 bytes, and GlobalSize will return 13.

We got a bug report that "Copy and Paste don't work" from a user with Windows 98. As you see from the screenshot above, I've got a Windows Me VM set up with VB6. Stepping through the code in the debugger I noticed the GlobalSize call, and remembered from the days of Win 95 that GlobalSize used to return the size of the actual allocated block, which was larger than you allocated and usually a multiple of 64. This led to the bug.

Now, the programmer at Microsoft who changed the behavior of GlobalSize probably thought he wasn't breaking anything. The documentation for the function "GlobalSize" says clearly, "the size of a memory block may be larger than the size requested when the memory was allocated." In fact the Microsoftee probably thought it was a minor and harmless improvement to GlobalSize to have it always return the size that you requested. Obviously any old code, that doesn't trust the return value from GlobalSize, will continue to work. So why not improve the function?

Not every programmer studies every line of the documentation for every function they use, and if the code is working, they tend to move on to something else. And it's not like everything is documented all that well -- the type of minutiae that I'm talking about here rarely gets discussed in documentation. And this is where you get these issues. The rest of the world discovered this problem, long the bane of WINE programmers, when the second web browser shipped, and suddenly everybody was noticing how the bugs they had counted on to make their pages look kewl weren't there any more. As we speak zillions of HTMLers are bemoaning the fact that IE6 now adheres to the standard for what <CENTER> does to the text inside a table, and so their pages that looked left-justified look like wedding invitations in IE6.

How did we get into this mess? Larry Wall famously said, "People understand instinctively that the best way for computer programs to communicate with each other is for each of the them to be strict in what they emit, and liberal in what they accept." I think that the evolution of HTML has proven that this isn't such a great idea. In fact, the stricter the API is about its input, the more likely the code is going to work in funny situations. The designers of Java got it right when they decided that nothing about the Java spec should leave any choice to the compiler developers (at least, not in the gratuitous way that C did, where the size of basic data types was not fixed). A better quote comes from Russian Field Marshal Suvorov: "A hard drill makes an easy battle." You want your compiler and your development environment to be as strict as possible; you want it to literally generate random return values for GlobalSize so that you don't get into the habit of counting on something that won't be there everywhere; you want to use French international settings on Chinese Windows 2000 with an absurd color scheme, DVORAK keyboard, trackball, 640x480 VGA mode, and huge ugly fonts on your development system so that you remember to bake in the code that adjusts for all these things. Then your application will be buff and strong and it will laugh in the face of wimpy problems like people who use commas instead of dots as the decimal. Ha. I eat commas for breakfast, your code will say, with a Russian accent.

Anyway, this is what it takes to get software that works on hundreds of millions of computers. Those of you who develop apps that only have to run on one computer or in a controlled environment have it easy, but you're getting flabby. One of these days you'll need to get it to run on a second computer and you'll need to pull an all nighter, installing a complete development environment on that computer and debugging for two hours, before you discover that you didn't account for the possibility of spaces in the installation file path because the first computer didn't have them.

Hopefully in the future the concept of a Virtual Machine, whether it's Java, .NET, or something else, will alleviate this pain, but we ain't there yet. For now I'm happy that with ten minutes of debugging, I can make my app work for all the people who like pink on orange text and set up Windows accordingly. We spent probably 3 weeks of work, out of a 1 year development cycle, fixing configuration bugs. A small price to pay to increase the size of our potential customer base from just US Windows 2000 to the entire universe of NT 4, 95, 98, Me, and XP, worldwide. Cool.


Advice for Computer Science College Students

Sunday, January 02, 2005

Despite the fact that it was only a year or two ago that I was blubbering about how rich Windows GUI clients were the wave of the future, college students nonetheless do occasionally email me asking for career advice, and since it's recruiting season, I thought I'd write up my standard advice which they can read, laugh at, and ignore.

Most college students, fortunately, are brash enough never to bother asking their elders for advice, which, in the field of computer science, is a good thing, because their elders are apt to say goofy, antediluvian things like "the demand for keypunch operators will exceed 100,000,000 by the year 2010" and "lisp careers are really very hot right now."

I, too, have no idea what I'm talking about when I give advice to college students. I'm so hopelessly out of date that I can't really figure out AIM and still use (horrors!) this quaint old thing called "email" which was popular in the days when music came on flat round plates called "CDs."

So you'd be better off ignoring what I'm saying here and instead building some kind of

Nevertheless.

If you enjoy programming computers, count your blessings: you are in a very fortunate minority of people who can make a great living doing work they enjoy. Most people aren't so lucky. The very idea that you can "love your job" is a modern concept. Work is supposed to be something unpleasant you do to get money to do the things you actually like doing, when you're 65 and can finally retire, if you can afford it, and if you're not too old and infirm to do those things, and if those things don't require reliable knees, good eyes, and the ability to walk twenty feet without being out of breath, etc.

What was I talking about? Oh yeah. Advice.

Without further ado, then, here are Joel's Seven Pieces of Free Advice for Computer Science College Students (worth what you paid for them):

  1. Learn how to write before graduating.
  2. Learn C before graduating.
  3. Learn microeconomics before graduating.
  4. Don't blow off non-CS classes just because they're boring.
  5. Take programming-intensive courses.
  6. Stop worrying about all the jobs going to India.
  7. No matter what you do, get a good summer internship.

Now for the explanations, unless you're gullible enough to do all that stuff just because I tell you to, in which case add: 8. Seek professional help for that self-esteem thing.

Learn how to write before graduating.

Would Linux have succeeded if Linus Torvalds hadn't evangelized it? As brilliant a hacker as he is, it was Linus's ability to convey his ideas in written English via email and mailing lists that made Linux attract a worldwide brigade of volunteers.

Have you heard of the latest fad, Extreme Programming? Well, without getting into what I think about XP, the reason you've heard of it is because it is being promoted by people who are very gifted writers and speakers.

Even on the small scale, when you look at any programming organization, the programmers with the most power and influence are the ones who can write and speak in English clearly, convincingly, and comfortably. Also it helps to be tall, but you can't do anything about that.

The difference between a tolerable programmer and a great programmer is not how many programming languages they know, and it's not

I won't hire a programmer unless they can write, and write well, in English. If you can write, wherever you get hired, you'll soon find that you're getting asked to write the specifications and that means you're already leveraging your influence and getting noticed by management.

Most colleges designate certain classes as "writing intensive," meaning, you have to write an awful lot to pass them. Look for those classes and take them! Seek out classes in any field that have weekly or daily written assignments.

Start a journal or weblog. The more you write, the easier it will be, and the easier it is to write, the more you'll write, in a virtuous circle.

Learn C before graduating

Part two: C. Notice I didn't say C++. Although C is becoming increasingly rare, it is still the lingua franca of working programmers. It is the language they use to communicate with one another, and, more importantly, it is much closer to the machine than "modern" languages that you'll be taught in college like ML, Java, Python, whatever trendy junk they teach these days. You need to spend at least a semester getting close to the machine or you'll never be able to create efficient code in higher level languages. You'll never be able to work on compilers and operating systems, which are some of the best programming jobs around. You'll never be trusted to create architectures for large scale projects. I don't care how much you know about continuations and closures and exception handling: if you can't explain why while (*s++ = *t++); copies a string, or if that isn't the most natural thing in the world to you, well, you're programming based on superstition, as far as I'm concerned: a medical doctor who doesn't know basic anatomy, passing out prescriptions based on what the pharma sales babe said would work.

Learn microeconomics before graduating

Super quick review if you haven't taken any economics courses: econ is one of those fields that starts off with a bang, with many

Why should CS majors learn econ? Because a programmer who understands the fundamentals of business is going to be a more valuable programmer, to a business, than a programmer who doesn't. That's all there is to it. I can't tell you how many times I've been frustrated by programmers with crazy ideas that make sense in code but don't make sense in capitalism. If you understand this stuff, you're a more valuable programmer, and you'll get rewarded for it, for reasons which you'll also learn in micro.

Don't blow off non-CS classes just because they're boring.

Blowing off your non-CS courses is a great way to get a lower GPA.

Never underestimate how big a deal your GPA is. Lots and lots of recruiters and hiring managers, myself included, go straight to the GPA when they scan a resume, and we're not going to apologize for it. Why? Because the GPA, more than any other one number, reflects the sum of what dozens of professors over a long period of time in many different situations think about your work. SAT scores? Ha! That's one test over a few hours. The GPA reflects hundreds of papers and midterms and classroom participations over four years. Yeah, it's got its problems. There has been grade inflation over the years. Nothing about your GPA says whether you got that GPA taking easy classes in home economics at Podunk Community College or taking graduate level Quantum Mechanics at Caltech. Eventually, after I screen out all the 2.5 GPAs from Podunk Community, I'm going to ask for transcripts and recommendations. And then I'm going to look for consistently high grades, not just high grades in computer science.

Why should I, as an employer looking for software developers, care about what grade you got in European History? After all, history is boring. Oh, so, you're saying I should hire you because you don't work very hard when the work is boring? Well, there's boring stuff in programming, too. Every job has its boring moments. And I don't want to hire people that only want to do the fun stuff.

I took this course in college called Cultural Anthropology because I figured, what the heck, I need to learn something about anthropology, and this looked like an interesting survey course.

Interesting? Not even close! I had to read these incredibly monotonous books about Indians in the Brazilian rain forest and Trobriand Islanders, who, with all due respect, are not very interesting to me. At some point, the class was so incredibly wearisome that I longed for something more exciting, like

I got an A. And if I could do it, you can do it.

Take programming-intensive courses.

I remember the exact moment I vowed never to go to graduate school.

It was in a course on Dynamic Logic, taught by the dynamic Lenore Zuck at Yale, one of the brightest of an array of very bright CS faculty.

Now, my murky recollections are not going to do proper credit to this field, but let me muddle through anyway. The idea of Formal Logic is that you prove things are true because other things are true. For example thanks to Formal Logic, "Everyone who gets good grades will get hired" plus "Johnny got good grades" allows you to discover the new true fact, "Johnny will get hired." It's all very quaint and it only takes ten seconds for a deconstructionist to totally tear apart everything useful in Formal Logic so you're left with something fun, but useless.

Now, dynamic logic is the same thing, with the addition of time. For example, "after you turn the light on, you can see your shoes" plus "The light went on in the past" implies "you can see your shoes."

Dynamic Logic is appealing to brilliant theoreticians like Professor Zuck because it holds up the hope that you might be able to formally prove things about computer programs, which could be very useful, if, for example, you could formally prove that the Mars Rover's flash card wouldn't overflow and cause itself to be rebooted again and again all day long when it's supposed to be driving around the red planet looking for Marvin the Martian.

So in the first day of that class, Dr. Zuck filled up two entire whiteboards and quite a lot of the wall next to the whiteboards proving that if you have a light switch, and the light was off, and you flip the switch, the light will then be on.

The proof was insanely complicated, and very error-prone. It was harder to prove that the proof was correct than to convince yourself of the fact that switching a light switch turns on the light. Indeed the multiple whiteboards of proof included many skipped steps, skipped because they were too tedious to go into formally. Many steps were reached using the long-cherished method of Proof by Induction, others by Proof by Reductio ad Absurdum, and still others using Proof by Graduate Student.

For our homework, we had to prove the converse: if the light was off, and it's on now, prove that you flipped it.

I tried, I really did.

I spent hours in the library trying.

After a couple of hours I found a mistake in Dr. Zuck's original proof which I was trying to emulate. Probably I copied it down wrong, but it made me realize something: if it takes three hours of filling up blackboards to prove something trivial, allowing hundreds of opportunities for mistakes to slip in, this mechanism would never be able to prove things that are interesting.

Not that that matters to dynamic logicians: they're not in it for useful, they're in it for tenure.

I dropped the class and vowed never to go to graduate school in Computer Science.

The moral of the story is that computer science is not the same as software development. If you're really really lucky, your school might have a decent software development curriculum, although, they might not, because elite schools think that teaching practical skills is better left to the technical-vocational institutes and the prison rehabilitation programs. You can learn mere programming anywhere. We are Yale University, and we Mold Future World Leaders. You think your $160,000 tuition entititles you to learn about while loops? What do you think this is, some fly-by-night Java seminar at the Airport Marriott? Pshaw.

The trouble is, we don't really have professional schools in software development, so if you want to be a programmer, you probably majored in Computer Science. Which is a fine subject to major in, but it's a different subject than software development.

If you're lucky, though, you can find lots of programming-intensive courses in the CS department, just like you can find lots of courses in the History department where you'll write enough to learn how to write. And those are the best classes to take. If you love programming, don't feel bad if you don't understand the point of those courses in lambda calculus or linear algebra where you never touch a computer. Look for the 400-level courses with Practicum in the name. This is an attempt to hide a useful (shudder) course from the Liberal Artsy Fartsy Administration by dolling it up with a Latin name.

Stop worrying about all the jobs going to India.

Well, OK, first of all, if you're already in India, you never really had to worry about this, so don't even start worrying about all the jobs going to India. They're wonderful jobs, enjoy them in good health.

But I keep hearing that enrollment in CS departments is dropping perilously, and one reason I hear for it is "students are afraid to go into a field where all the jobs are going to India." That's so wrong for so many reasons. First, trying to choose a career based on a current business fad is foolish. Second, programming is incredibly good training for all kinds of fabulously interesting jobs, such as business process engineering, even if every single programming job does go to India and China. Third, and trust me on this, there's still an incredible shortage of the really good programmers, here and in India. Yes, there are a bunch of out of work IT people making a lot of noise about how long they've been out of work, but you know what? At the risk of pissing them off, really good programmers do have jobs. Fourth, you got any better ideas? What are you going to do, major in History? Then you'll have no choice but to go to law school. And there's one thing I do know: 99% of working lawyers hate their jobs, hate every waking minute of it, and they're working 90 hour weeks, too. Like I said: if you love to program computers, count your blessings: you are in a very fortunate minority of people who can make a great living doing work they love.

Anyway, I don't think students really think about this. The drop in CS enrollment is merely a resumption of historically normal levels after a big bubble in enrollment caused by dotcom mania. That bubble consisted of people who didn't really like programming but thought the sexy high paid jobs and the chances to IPO at age 24 were to be found in the CS department. Those people, thankfully, are long gone.

No matter what you do, get a good summer internship.

Smart recruiters know that the people who love programming wrote a database for their dentist in 8th grade, and taught at computer camp for three summers before college, and built the content management system for the campus newspaper, and had summer internships at software companies. That's what they're looking for on your resume.

If you enjoy programming, the biggest mistake you can make is to take any kind of job--summer, part time, or otherwise--that is not a programming job. I know, every other 19-year-old wants to work in the mall folding shirts, but you have a skill that is incredibly valuable even when you're 19, and it's foolish to waste it folding shirts. By the time you graduate, you really should have a resume that lists a whole bunch of programming jobs. The A&F graduates are going to be working at Enterprise Rent-a-Car "helping people with their rental needs." (Except for Tom Welling. He plays Superman on TV.)

To make your life really easy, and to underscore just how completely self-serving this whole essay is, my company, Fog Creek Software, has summer internships in software development that look great on resumes. "You will most likely learn more about software coding, development, and business with Fog Creek Software than any other internship out there," says Ben, one of the interns from last summer, and not entirely because I sent a goon out to his dorm room to get him to say that. The application deadline is February 1st. Get on it.

If you follow my advice, you, too, may end up selling stock in Microsoft way too soon, turning down jobs at Google because you want your own office with a door, and other stupid life decisions, but they won't be my fault. I told you not to listen to me.


Anonymous Response

Tuesday, July 25, 2000

An anonymous response from inside Microsoft on my article Microsoft Goes Bonkers writes:

It's not just you...many of us at MS don't even begin to understand what .NET is (and I even work on Passport, the shining example of a "web service"). Management spent nearly a year explaining how everyone needed to focus on NGWS and how we could all fit into the vision - without ever describing the goal. It was the proverbial answer in search of a question. All of a sudden it has a new name, seemingly an attempt to hide the fact that it still has no body. And to make things worse, they throw in a brand-new programming language which is really nothing more but a copy of java which is unfinished, hasn't been tested for five years, and lacks a large standard library.
I've asked around how this new .NET plan differs from everything we've been working on the past two years and haven't been given a decent answer.
"What's a web service?"
"Look at Passport! THAT'S a service!" (they love saying that one)
"You mean a web service is just another site that you visit that drops encrypted cookies on your machine?"
"No no no. It enables people to work together! It empowers companies to share information!"
"You mean it lets partners access information that _WE_ maintain in a central repository? How is that different from any other website?"
"We allow users to interact with the service! It's not a static vision of the web! It's a two-way collaboration!"
"You mean users fill out forms to create an account. And then we let them access it. What is new about that?"
The answers just kept getting more incomprehensible.
I don't think we know what .NET is. It is becoming all things to all people and the vision continues to grow. Every product that ships in the next two years seems to be added under the umbrella. I'm amazed more people haven't laid into us for being so vague. Perhaps .NET is nothing more than a hopeful solution to a deeper problem...a lack of focus in the company - a missing goal and rallying point. Unfortunately, we need a lot more than hand-waving right now.
Thanks for the intriguing posts!

Advertainment! Do you need to control a computer remotely, even when firewalls get in the way? My company's latest product, Fog Creek Copilot, is a remote control system that requires no setup, no configuration, and works even if both users are behind firewalls. It's designed to make remote tech support easy.

Enter your email address to receive a (very occasional) email whenever I write a major new article. You can unsubscribe at any time, of course.
Email:


Another Business Model That Doesn't Seem to Work

Wednesday, October 25, 2000

I used to have Atomz.com as my search provider. That means that when you typed a search request in that little search box in the margin, it actually sent the request to Atomz.com. They were responsible for indexing my site once a week and displaying the results.

Atomz.com is "free", with a catch: as soon as you have more than 500 pages on your site, they send you a nice little email:

I just wanted to take this opportunity to remind you that your website has exceeded the 500 page document limit for the free account that you currently have. Did you realize that 100% of your site is not currently being indexed? Only the FIRST 500 PAGES are being indexed. Is this an important concern to you and the services that you provide to others? If yes, upgrading to an atomz.com prime service will solve these issues.

Nowhere in the email does it tell you how much it's going to cost for the upgrade. A bit of searching on their web site gave me the answer: $100 per quarter. Wow. That's expensive.

Search on my site is not super popular; I get about 60 searches a week (out of about 35,000 page views). But I think it's an important function. And I hate being tricked into upgrading from a 'free' service to a 'premium' service.

Atomz.com is hoping to achieve what I call 'stealth lock-in'. They don't even want to you realize that you're becoming dependent on a service. In the future, when you're hooked, they can start charging you a bundle, and you'll pay them just to avoid the cost of switching. There are a lot of Internet companies that are trying this approach. Juno constantly bombards its unpaying members to upgrade to the services that cost money. All the various xdrive/idrive/freedrive/mydrive clones have this business model. My buddy Elan's company earthnoise, which gives you 50M of online video storage free, tries to upgrade you to the premium service at $4.99 a month when you realize that 50M is not that much online video.

The question here is: how much lock-in do these companies really have? That's what is going to determine whether their plans work or fail. Juno has a lot, because it's an email service, and if you switch your email provider, you have to tell all your friends about the new email address. If you printed a Juno email address on your business card and handed it out to prospective clients, you're going to be scared to switch away. And indeed, Juno has been reasonably successful at getting people to upgrade.

All the free-hard-drive-on-the-net companies have zero lock in. They are going to have a problem. Besides the fact that there is almost no barrier to entry, incredible competition, and the service is stupid to begin with, these companies are simply not going to succeed, period.

Now, back to Atomz.com, my old search service. Lock-in? Well, I'm afraid not. It took me 20 minutes to switch over to Google. Kiss them goodbye.


Are the Groove Designers Architecture Astronauts?

Sunday, April 22, 2001

George Moromisato at Groove Networks writes:

I read "Joel on Software" more or less religiously and I'm always intrigued by your insights. Your column on Architecture Astronauts was no different, except for your dismissal of Groove as software that "miss[ed] the point" of Napster's example. I hope to take a few moments of your time to convince you to give Groove another chance.

Like Napster, we also consider peer-to-peer to be a means to an end, not the end in itself. For Groove, the end goal is not to search for music on the Internet (as cool as that is); Groove's end goal is to allow people to interact and share information on the Internet without regards to time, distance, or corporate infrastructure.

The last point is worth diving on: Today, if I want to share information with someone outside my corporate infrastructure I have three plausible choices: 1) use email, which is not very interactive, has very little persistence, and is often insecure; 2) use an internal web server, which requires me to ask my IT administrator to either open up a hole in the firewall or to deploy some section on our extranet; or 3) use a public web server, which now is privy to my corporate data and may take it with them when they go under. The cool thing about Groove, and one of the problems that we were trying to solve, is that I can use it like email to communicate with someone outside my infrastructure (they can download the client for free) and I don't have to worry about a server (we are peer-to-peer when appropriate) or a firewall (we go through firewalls) or security (since Groove communications are by invitation only and heavily encrypted). The best part, IMHO, is that, while we provide some basic tools to interact inside a Groove space (sharing files, doodling, voice chat, forums, etc.), we have also built a platform that allows third parties to extend Groove in whatever direction they want, all without having to ask us to change our servers or upgrade the core product!

[pause while I stop hyperventilating]

The bottom line is that I think it was unfair of you to consider us Architecture Astronauts. It is true that we do not solve the problem that Napster was trying to solve, but I think we have been successful at solving the problem that we set out to solve. Also, to be fair, we have managed to ship a product (and even sell it to people) unlike some of the Architecture Astronauts that you are thinking of.

Anyway, I hope I haven't completely wasted your time. Thanks for your columns and good luck with everything.

George Moromisato
Chief Product Designer
Groove Networks

My reply:

I like Groove, I think it's a nice architecture. I think in many ways it's Notes done right, eliminating one of the biggest problems with Notes -- the server bottleneck which crippled large Notes installations.

On the other hand, Notes was the most misunderstood product in history :) It was being sold as a groupware platform, but in reality, it was being bought (and used) as an email program. A few people used the discussion groups and document databases, and an even smaller number actually created applications on top of Notes, but the reason companies with 50,000 people bought Notes was really for email.

Which, unfortunately, didn't work quite well enough in Notes. At Bankers Trust in the mid-90's, management ruled that no new Notes accounts could be created, because of an incident where Notes replication between servers managed to tie up the company's crucial transatlantic link thus cutting London off from New York for an entire trading day. As a result Bankers Trust was in the awkward position of ruling "no new Notes accounts!"

At Viacom a couple of years later, Notes was shelfware because the email component was too hard for most people to figure out. Everybody in the company had AOL accounts which they expensed because they could figure that out.

Note that none of this is necessarily a bad thing from a company perspective: Notes sold scrillions of copies and was considered a raging success. In theory, Groove is even cooler, because it's Notes without the server headaches.

Anyway, now we're talking about architecture. Let's talk about features. The applets you ship with are all spiffy but not ultra-compelling, because you can often get the same functionality elsewhere. And it's the applets that are going to sell Groove, not the architecture. If you don't believe me look at NeXT and Be ... you can build the best dang computer in the world with a killer architecture, but if it doesn't run Microsoft Excel, nobody wants one. When you say "it's a platform, third parties will build things," you have to explain what KIND of things and why THEY will be killer apps that sell the architecture. If you keep hearing the same examples, you're in deep trouble -- like the idiot WAP people who talk endlessly about how "you'll walk by a starbucks and the GPS in your phone will coordinate to beam you a coupon for that starbucks." I've heard this same example a zillion times from "location based wireless" architecture astronauts and it amuses me, because it solves the one problem that coffee shops DON'T have, namely, advertising to people who are standing right in front of the store! :)

So... yes, you shipped a product, and sold it to people, it's great, I believe you. But if you want it to be The Next Great Thing it has to be more than architecture, it has to enable things that people really need.


Ask Joel

Wednesday, August 22, 2001

... in which I respond to some interesting email I've gotten lately. All the incoming email has been edited for brevity.

Charles Lin writes:

I teach a C++ computer science course here at University of Maryland at College Park....

Presume you are the czar of University of Maryland Computer Science (you've just dreamed about this, haven't you?), and say "things need to be changed -- we need to teach computer science the right way", what would you do? What do you think are the important things students need to learn if they are to become good C++ programmers.

Off of the top of my head, some of these topics are:

  1. the relational model and modern DBMSs
  2. user interface design and usability
  3. techniques for coding on large teams
  4. software design and writing functional specifications
  5. debugging
  6. Windows and COM programming
  7. sockets programming and common Internet protocols
  8. database-backed web site development
  9. writing! Because great programmers are great communicators.

Dan Green writes:

I read sometime back that you were going shared sourced (for want of a better term) with FogBUGZ. I'm going to be wanting to do similar on a product of mine but I'm not sure how best to word the license. Would you mind sharing the FogBugz license, so that I can learn from it?

Sure. It's here. I suggest you get a lawyer, though, to review your license. (If you have Errors and Omissions insurance, whatever the heck that is, you are probably required to do so).

Paul Wolpe writes:

I graduated from Carnegie Mellon in May and was happily planning on working for Loudcloud in New York City this fall. It seemed like the best of both worlds; I could work for a tech company and I wouldn't need to live in the suburbs. Well, due to "unfavorable market conditions" I am now without a job and faced with what appears to be two options: go work for Microsoft in Redmond or work for some financial company in New York City. I have to choose between a great job in a not so great location or a great location with a not so great job. I am not asking for a job (but I would gladly accept if you are offering!). I want to find out if there are any interesting tech companies in New York (and even better if they are hiring). So far, I have found a total of maybe 3 companies, and that includes Fog Creek.

Oooo. Uh-oh. I have two pieces of bad news for ya. No, wait, three pieces. Number one: there aren't that many technology companies where you can do R&D in New York City. Every local programmer I know complains all the time about how hard it is to get interesting software jobs that aren't at a bank. That's one of the reasons I created Fog Creek. Which brings us to bad news number two: Fog Creek is not hiring, because we can't afford to right now, which is caused by the dot-com crash, which is bad news number three: there's a bit of a recession going on right now, especially in the tech sector... which means that almost nobody is hiring.

But there is good news, which is that every programmer I know has some job. Maybe it's not their dream job, but it pays the rent.

The good thing about working for Microsoft is that Microsoft is one company that really knows how to make software, and you'll learn a lot that will help you wherever you go. And the good thing about living in New York City is, well, OK, I don't have to explain that. I think in your place I would come to the city and spend some time hopping around, waiting for something interesting to show up. (Pay your dues at the banks and media companies like the rest of us!) Oh, and email me your resume anyway, you never know what might come up.

Gabriel Krupa writes:

I have been a fan of yours for a while and would very much like to start my own company, modeled after Fog Creek (on the other side of the country). Given the business climate lately, I'm hesitant to leave "subsistence programming" (and give it a go with my own savings) until I plan, scheme and "strategize" myself into a false sense of security. I understand the importance of leveraging work to generate money, lest I spend the rest of my life running a consulting firm. Unfortunately, it's not easy to pay someone else's salary from my margins unless they're generating part of that revenue for the company-- at least with my hourly rate. Thus, I'm still a company of one with less time than I'd like to devote to building the company.

This is where I get to the point. I've read your admissions of how much it takes to run Fog Creek on a monthly basis and that you're making most of that through consulting. I'd like to find out about Fog Creek's shrink-wrap strategy and results to date. If you'd rather not share this information, I understand.

For instance, I'd like to know:

  • how much revenue you generate from FogBugz and expect from CityDesk

    FogBUGZ is not yet at the point where it's paying anybody's salary, but it does cover virtually all of our non-salary expenses (rent, T1, etc) and then some. Which is nice. Yesterday we sold $5700 worth in one day. Whoo hoo! Your mileage may vary.

    We expect CityDesk to have a much wider potential audience than FogBUGZ, so we're hoping that once CityDesk hits its stride we won't need any consulting revenue to support ourselves.

  • what sort of growth you've seen and expect(ed) in sales

    It's probably too early to see a pattern, but there has been slow but steady growth as the word spreads. We are not doing any active marketing on FogBUGZ at this point, mainly because we want to conserve any marketing dollars we have for CityDesk.

    Even without funding, we hope to be able to get to the point where we can demonstrate that (a) we're profitable and secure and (b) x dollars spent on marketing results in x times y dollars in revenue. At this point, if y > 1, we can make a strong argument to raise money for marketing without giving up very much equity... or we can go slowly and simply reinvest internally.

  • how you choose your product lines (e.g., why you chose to make FogBugz and CityDesk instead of other products (user demand and market conditions being roughly equal))

    FogBUGZ was something we had, so it was a no-brainer to ship it: both to get some cash and to get some practice selling software. (And to get the people off our backs who kept asking us if they could buy it.) It has more than paid for its own R&D costs, so now it feels like one of those mystical things you hear about on late-night TV that you set up and get a nice check every month in exchange for doing nothing. (We do still plan several more major releases of FogBUGZ so it's not exactly nothing but you get the idea).

    CityDesk will fill a very major vacuum in the desktop software marketplace, it's a great product, and everybody who's seen it has said "I gotta get me one of those." So we're working day and night trying to get the beta out the door.

  • if you feel your reputation and product quality are enough to gain market share in non-virgin markets (i.e., defect tracking) or what Fog Creek's strategy is

    There are about 1000 bug tracking packages on the market (from companies who are probably selling it for the same reason as we are -- because they have it.) So we didn't know if we would sell any copies of FogBUGZ. As it turns out, the FogBUGZ philosophy of painless bug tracking has hit a nerve and the product has exceeded our expectations and then some. People are also happy because we're a small company that provides fanatical customer service... when you find a bug and call our toll-free number, you'll speak to a programmer and probably get a bug fix shipped to you that day.

  • whether or not you're presently concentrating on non-consulting revenue

    We're trying to avoid taking new clients for a while, since we don't actually need the money and would rather have CityDesk done right now. But this is something we reevaluate constantly.

Ted Graham Writes:

Since you are an ex-microsoftie, I have a question for you: does Microsoft use VSS for their projects? I can't see how they would, it doesn't seem to have the features needed to support multiple versions. I'm a consultant on a fairly large VC++ project on Win2K and we are about to make our first shipment to the end users. I've been looking for a way to do branching like I'm used to in PVCS, but not finding anything. It seems like it has labels available, but we can't branch an individual file off and label the branch. Any ideas?

I can't really speak for what Microsoft uses; I haven't been there in years and SourceSafe didn't exist when I was there. But I have heard only bad things about it. What I've heard from other programmers is that it's not so strong in the data reliability department. Oops.

Do yourself a favor - use CVS like everyone else; it's simple, reliable, ubiquitous, and rock-solid.


Auto motion in Excel?

Friday, May 12, 2000

Hanan Cohen writes:

A long time ago. in the 80's of the last century, I have used a spreadsheet program called Saturn on a DEC PDP/11 running RSTS-E. It had a wonderful feature which I can't find in Excel. It was called "auto motion". If you chose this feature, you could enter data in a cell, press any arrow key, enter the data in the next cell and when pressing Enter, the cursor would move to the next cell in the same direction of the first movement. You could enter data from the bottom to the top, right to left of any direction you chose. In Excel, you can either choose "top to bottom" or "left to right".
If you still have friends at the Excel team, tell them about this feature. It was wonderful and a "must have".
My response:
Ah, you're trying to trick me into recommending a feature that I don't approve of :)
Here's why I don't approve: it's confusing. Many people will see this behavior and they won't realize *why* the Enter key suddenly wants to go UP instead of DOWN. By then they may have forgotten that they previously pressed the up arrow.
It's important that things (especially the enter key) behave in the same way instead of getting into invisible modes. Nothing on the screen tells you why Enter is going up now, and you may just decide it's a bug and get frustrated. You'll get especially frustrated if you look away from the screen (at the piece of paper with all the numbers you are typing in) and type in a bunch of numbers, only to look back and find out that not only are they in the wrong cells, but you have accidentally overwritten something interesting you used to have there.

Advertainment! Do you need to control a computer remotely, even when firewalls get in the way? My company's latest product, Fog Creek Copilot, is a remote control system that requires no setup, no configuration, and works even if both users are behind firewalls. It's designed to make remote tech support easy.

Enter your email address to receive a (very occasional) email whenever I write a major new article. You can unsubscribe at any time, of course.
Email:


Back to Basics

Tuesday, December 11, 2001

We spend a lot of time on this site talking about exciting Big Picture Stuff like .NET versus Java, XML strategy, Lock-In, competitive strategy, software design, architecture, and so forth. All this stuff is a layer cake, in a way. At the top layer, you've got software strategy. Below that, we think about architectures like .NET, and below that, individual products: software development products like Java or platforms like Windows.

Go lower on the cake, please. DLLs? Objects? Functions? No! Lower! At some point you're thinking about lines of code written in programming languages.

Still not low enough. Today I want to think about CPUs. A little bit of silicon moving bytes around. Pretend you are a beginning programmer. Tear away all that knowledge you've built up about programming, software, management, and get back to the lowest level Von Neumann fundamental stuff. Wipe J2EE out of your mind for a moment. Think Bytes.

Why are we doing this? I think that some of the biggest mistakes people make even at the highest architectural levels come from having a weak or broken understanding of a few simple things at the very lowest levels. You've built a marvelous palace but the foundation is a mess. Instead of a nice cement slab, you've got rubble down there. So the palace looks nice but occasionally the bathtub slides across the bathroom floor and you have no idea what's going on.

So today, take a deep breath. Walk with me, please, through a little exercise which will be conducted using the C programming language.

Remember the way strings work in C: they consist of a bunch of bytes followed by a null character, which has the value 0. This has two obvious implications:

  1. There is no way to know where the string ends (that is, the string length) without moving through it, looking for the null character at the end.
  2. Your string can't have any zeros in it. So you can't store an arbitrary binary blob like a JPEG picture in a C string.

Why do C strings work this way? It's because the PDP-7 microprocessor, on which UNIX and the C programming language were invented, had an ASCIZ string type. ASCIZ meant "ASCII with a Z (zero) at the end."

Is this the only way to store strings? No, in fact, it's one of the worst ways to store strings. For non-trivial programs, APIs, operating systems, class libraries, you should avoid ASCIZ strings like the plague. Why?

Let's start by writing a version of the code for strcat, the function which appends one string to another.

void strcat( char* dest, char* src )
{
while (*dest) dest++;
while (*dest++ = *src++);
}

Study the code a bit and see what we're doing here. First, we're walking through the first string looking for its null-terminator. When we find it, we walk through the second string, copying one character at a time onto the first string.

This kind of string handling and string concatenation was good enough for Kernighan and Ritchie, but it has its problems. Here's a problem. Suppose you have a bunch of names that you want to append together in one big string:

char bigString[1000]; /* I never know how much to allocate... */
bigString[0] = '\0';
strcat(bigString,"John, ");
strcat(bigString,"Paul, ");
strcat(bigString,"George, ");
strcat(bigString,"Joel ");

This works, right? Yes. And it looks nice and clean.

What is its performance characteristic? Is it as fast as it could be? Does it scale well? If we had a million strings to append, would this be a good way to do it?

No. This code uses the Shlemiel the painter's algorithm. Who is Shlemiel? He's the guy in this joke:

Shlemiel gets a job as a street painter, painting the dotted lines down the middle of the road. On the first day he takes a can of paint out to the road and finishes 300 yards of the road. "That's pretty good!" says his boss, "you're a fast worker!" and pays him a kopeck.

The next day Shlemiel only gets 150 yards done. "Well, that's not nearly as good as yesterday, but you're still a fast worker. 150 yards is respectable," and pays him a kopeck.

The next day Shlemiel paints 30 yards of the road. "Only 30!" shouts his boss. "That's unacceptable! On the first day you did ten times that much work! What's going on?"

"I can't help it," says Shlemiel. "Every day I get farther and farther away from the paint can!"

(For extra credit,

How do we fix this? A few smart C programmers implemented their own mystrcat as follows:

char* mystrcat( char* dest, char* src )
{
 while (*dest) dest++;
while (*dest++ = *src++);
return --dest;
}

What have we done here? At very little extra cost we're returning a pointer to the end of the new, longer string. That way the code that calls this function can decide to append further without rescanning the string:

char bigString[1000]; /* I never know how much to allocate... */
char *p = bigString;
bigString[0] = '\0';
p = mystrcat(p,"John, ");
p = mystrcat(p,"Paul, ");
p = mystrcat(p,"George, ");
p = mystrcat(p,"Joel ");

This is, of course, linear in performance, not n-squared, so it doesn't suffer from degradation when you have a lot of stuff to concatenate.

The designers of Pascal were aware of this problem and "fixed" it by storing a byte count in the first byte of the string. These are called Pascal Strings. They can contain zeros and are not null terminated. Because a byte can only store numbers between 0 and 255, Pascal strings are limited to 255 bytes in length, but because they are not null terminated they occupy the same amount of memory as ASCIZ strings. The great thing about Pascal strings is that you never have to have a loop just to figure out the length of your string. Finding the length of a string in Pascal is one assembly instruction instead of a whole loop. It is monumentally faster.

The old Macintosh operating system used Pascal strings everywhere. Many C programmers on other platforms used Pascal strings for speed. Excel uses Pascal strings internally which is why strings in many places in Excel are limited to 255 bytes, and it's also one reason Excel is blazingly fast.

For a long time, if you wanted to put a Pascal string literal in your C code, you had to write:

char* str = "\006Hello!";

Yep, you had to count the bytes by hand, yourself, and hardcode it into the first byte of your string. Lazy programmers would do this, and have slow programs:

char* str = "*Hello!";
str[0] = strlen(str) - 1;

Notice in this case you've got a string that is null terminated (the compiler did that) as well as a Pascal string. I used to call these fucked strings because it's easier than calling them null terminated pascal strings but this is a rated-G channel so you will have use the longer name.

I elided an important issue earlier. Remember this line of code?

char bigString[1000]; /* I never know how much to allocate... */

Since we're looking at the bits today I shouldn't have ignored this. I should have done this correctly: figured out how many bytes I needed and allocated the right amount of memory.

Shouldn't I have?

Because otherwise, you see, a clever hacker will read my code and notice that I'm only allocating 1000 bytes and hoping it will be enough, and they'll find some clever way to trick me into strcatting a 1100 byte string into my 1000 bytes of memory, thus overwriting the stack frame and changing the return address so that when this function returns, it executes some code which the hacker himself wrote. This is what they're talking about when they say that a particular program has a buffer overflow susceptibility. It was the number one cause of hacks and worms in the olden days before Microsoft Outlook made hacking easy enough for teenagers to do.

OK, so all those programmers are just lame-asses. They should have figured out how much memory to allocate.

But really, C does not make this easy on you. Let's go back to my Beatles example:

char bigString[1000]; /* I never know how much to allocate... */
char *p = bigString;
bigString[0] = '\0';
p = mystrcat(p,"John, ");
p = mystrcat(p,"Paul, ");
p = mystrcat(p,"George, ");
p = mystrcat(p,"Joel ");

How much should we allocate? Let's try doing this The Right Way.

char* bigString;
int i = 0;
i = strlen("John, ")
+ strlen("Paul, ")
+ strlen("George, ")
+ strlen("Joel ");
bigString = (char*) malloc (i + 1);
/* remember space for null terminator! */
...

My eyes glazeth over. You're probably about ready to change the channel already. I don't blame you, but bear with me because it gets really interesting.

We have to scan through all the strings once just figuring out how big they are, then we scan through them again concatenating. At least if you use Pascal strings the strlen operation is fast. Maybe we can write a version of strcat that reallocates memory for us.

That opens another whole can of worms: memory allocators. Do you know how malloc works? The nature of malloc is that it has a long linked list of available blocks of memory called the free chain. When you call malloc, it walks the linked list looking for a block of memory that is big enough for your request. Then it cuts that block into two blocks -- one the size you asked for, the other with the extra bytes, and gives you the block you asked for, and puts the leftover block (if any) back into the linked list. When you call free, it adds the block you freed onto the free chain. Eventually, the free chain gets chopped up into little pieces and you ask for a big piece and there are no big pieces available the size you want. So malloc calls a timeout and starts rummaging around the free chain, sorting things out, and merging adjacent small free blocks into larger blocks. This takes 3 1/2 days. The end result of all this mess is that the performance characteristic of malloc is that it's never very fast (it always walks the free chain), and sometimes, unpredictably, it's shockingly slow while it cleans up. (This is, incidentally, the same performance characteristic of garbage collected systems, surprise surprise, so all the claims people make about how garbage collection imposes a performance penalty are not entirely true, since typical malloc implementations had the same kind of performance penalty, albeit milder.)

Smart programmers minimize the potential distruption of malloc by always allocating blocks of memory that are powers of 2 in size. You know, 4 bytes, 8 bytes, 16 bytes, 18446744073709551616 bytes, etc. For reasons that should be intuitive to anyone who plays with Lego, this minimizes the amount of weird fragmentation that goes on in the free chain. Although it may seem like this wastes space, it is also easy to see how it never wastes more than 50% of the space. So your program uses no more than twice as much memory as it needs to, which is not that big a deal.

Suppose you wrote a smart strcat function that reallocates the destination buffer automatically. Should it always reallocate it to the exact size needed? My teacher and mentor Stan Eisenstat suggests that when you call realloc, you should always double the size of memory that was previously allocated. That means that you never have to call realloc more than lg n times, which has decent performance characteristics even for huge strings, and you never waste more than 50% of your memory.

Anyway. Life just gets messier and messier down here in byte-land. Aren't you glad you don't have to write in C anymore? We have all these great languages like Perl and Java and VB and XSLT that never make you think of anything like this, they just deal with it, somehow. But occasionally, the plumbing infrastructure sticks up in the middle of the living room, and we have to think about whether to use a String class or a StringBuilder class, or some such distinction, because the compiler is still not smart enough to understand everything about what we're trying to accomplish and is trying to help us not write inadvertent Shlemiel the Painter algorithms.

Last week I wrote that you can't implement the SQL statement SELECT author FROM books fast when your data is stored in XML. Just in case everybody didn't understand what I was talking about, and now that we've been rolling around in the CPU all day, this assertion might make more sense.

How does a relational database implement SELECT author FROM books? In a relational database, every row in a table (e.g. the books table) is exactly the same length in bytes, and every fields is always at a fixed offset from the beginning of the row. So, for example, if each record in the books table is 100 bytes long, and the author field is at offset 23, then there are authors stored at byte 23, 123, 223, 323, etc. What is the code to move to the next record in the result of this query? Basically, it's this:

pointer += 100;

One CPU instruction. Faaaaaaaaaast.

Now lets look at the books table in XML.

<?xml blah blah>
<books>
<book>
<title>UI Design for Programmers</title>
<author>Joel Spolsky</author>
</book>
<book>
<title>The Chop Suey Club</title>
<author>Bruce Weber</author>
</book>
</books>

Quick question. What is the code to move to the next record?

Uh...

At this point a good programmer would say, well, let's parse the XML into a tree in memory so that we can operate on it reasonably quickly. The amount of work that has to be done here by the CPU to SELECT author FROM books will bore you absolutely to tears. As every compiler writer knows, lexing and parsing are the slowest part of compiling. Suffice it to say that it involves a lot of string stuff, which we discovered is slow, and a lot of memory allocation stuff, which we discovered is slow, as we lex, parse, and build an abstract syntax tree in memory. That assumes that you have enough memory to load the whole thing at once. With relational databases, the performance of moving from record to record is fixed and is, in fact, one CPU instruction. That's very much by design. And thanks to memory mapped files you only have to load the pages of disk that you are actually going to use. With XML, if you preparse, the performance of moving from record to record is fixed but there's a huge startup time, and if you don't preparse, the performance of moving from record to record varies based on the length of the record before it and is still hundreds of CPU instructions long.

What this means to me is that you can't use XML if you need performance and have lots of data. If you have a little bit of data, or if what you're doing doesn't have to be fast, XML is a fine format. And if you really want the best of both worlds, you have to come up with a way to store metadata next to your XML, something like Pascal strings' byte count, which give you hints about where things are in the file so that you don't have to parse and scan for them. But of course then you can't use text editors to edit the file because that messes up the metadata, so it's not really XML anymore.

For those three gracious members of my audience who are still with me at this point, I hope you've learned something or rethought something. I hope that thinking about boring first-year computer-science stuff like how strcat and malloc actually work has given you new tools to think about the latest, top level, strategic and architectural decisions that you make in dealing with technologies like XML. For homework, think about why Transmeta chips will always feel sluggish. Or why the original HTML spec for TABLES was so badly designed that large tables on web pages can't be shown quickly to people with modems. Or about why COM is so dang fast but not when you're crossing process boundaries. Or about why the NT guys put the display driver into kernelspace instead of userspace.

These are all things that require you to think about bytes, and they affect the big top-level decisions we make in all kinds of architecture and strategy. This is why my view of teaching is that first year CS students need to start at the basics, using C and building their way up from the CPU. I am actually physically disgusted that so many computer science programs think that Java is a good introductory language, because it's "easy" and you don't get confused with all that boring string/malloc stuff but you can learn cool OOP stuff which will make your big programs ever so modular. This is a pedagogical disaster waiting to happen. Generations of graduates are descending on us and creating Shlemiel The Painter algorithms right and left and they don't even realize it, since they fundamentally have no idea that strings are, at a very deep level, difficult, even if you can't quite see that in your perl script. If you want to teach somebody something well, you have to start at the very lowest level. It's like Karate Kid. Wax On, Wax Off. Wax On, Wax Off. Do that for three weeks. Then Knocking The Other Kid's Head off is easy.


Biculturalism

Sunday, December 14, 2003

By now, Windows and Unix are functionally more similar than different. They both support the same major programming metaphors, from command lines to GUIs to web servers; they are organized around virtually the same panoply of system resources, from nearly identical file systems to memory to sockets and processes and threads. There's not much about the core set of services provided by each operating system to limit the kinds of applications you can create.

What's left is cultural differences. Yes, we all eat food, but over there, they eat raw fish with rice using wood sticks, while over here, we eat slabs of ground cow on bread with our hands. A cultural difference doesn't mean that American stomachs can't digest sushi or that Japanese stomachs can't digest Big Macs, and it doesn't mean that there aren't lots of Americans who eat sushi or Japanese who eat burgers, but it does mean that Americans getting off the plane for the first time in Tokyo are confronted with an overwhelming feeling that this place is strange, dammit, and no amount of philosophizing about how underneath we're all the same, we all love and work and sing and die will overcome the fact that Americans and Japanese can never really get comfortable with each others' toilet arrangements.

What are the cultural differences between Unix and Windows programmers? There are many details and subtleties, but for the most part it comes down to one thing: Unix culture values code which is useful to other programmers, while Windows culture values code which is useful to non-programmers.

This is, of course, a major simplification, but really, that's the big difference: are we programming for programmers or end users? Everything else is commentary.

The frequently controversial Eric S. Raymond has just written a long book about Unix programming called The Art of UNIX Programming exploring his own culture in great detail. You can buy the book and read it on paper, or, if Raymond's politics are just too anti-idiotarian for you to consider giving him money, you can even read it online for free and rest assured that the author will not receive a penny for his hard work.

Let's look at a small example. The Unix programming culture holds in high esteem programs which can be called from the command line, which take arguments that control every aspect of their behavior, and the output of which can be captured as regularly-formatted, machine readable plain text. Such programs are valued because they can easily be incorporated into other programs or larger software systems by programmers. To take one miniscule example, there is a core value in the Unix culture, which Raymond calls "Silence is Golden," that a program that has done exactly what you told it to do successfully should provide no output whatsoever. It doesn't matter if you've just typed a 300 character command line to create a file system, or built and installed a complicated piece of software, or sent a manned rocket to the moon. If it succeeds, the accepted thing to do is simply output nothing. The user will infer from the next command prompt that everything must be OK.

This is an important value in Unix culture because you're programming for other programmers. As Raymond puts it, "Programs that babble don't tend to play well with other programs." By contrast, in the Windows culture, you're programming for Aunt Marge, and Aunt Marge might be justified in observing that a program that produces no output because it succeeded cannot be distinguished from a program that produced no output because it failed badly or a program that produced no output because it misinterpreted your request.

Similarly, the Unix culture appreciates programs that stay textual. They don't like GUIs much, except as lipstick painted cleanly on top of textual programs, and they don't like binary file formats. This is because a textual interface is easier to program against than, say, a GUI interface, which is almost impossible to program against unless some other provisions are made, like a built-in scripting language. Here again, we see that the Unix culture values creating code that is useful to other programmers, something which is rarely a goal in Windows programming.

Which is not to say that all Unix programs are designed solely for programmers. Far from it. But the culture values things that are useful to programmers, and this explains a thing or two about a thing or two.

Suppose you take a Unix programmer and a Windows programmer and give them each the task of creating the same end-user application. The Unix programmer will create a command-line or text-driven core and occasionally, as an afterthought, build a GUI which drives that core. This way the main operations of the application will be available to other programmers who can invoke the program on the command line and read the results as text. The Windows programmer will tend to start with a GUI, and occasionally, as an afterthought, add a scripting language which can automate the operation of the GUI interface. This is appropriate for a culture in which 99.999% of the users are not programmers in any way, shape, or form, and have no interest in being one.

There is one significant group of Windows programmers who are primarily coding for other programmers: the Windows team itself, inside Microsoft. The way they tend to do things is to create an API, callable from the C language, which implements the functionality, and then create GUI applications which call that API. Anything you can do from the Windows user interface can also be accomplished using a programming interface callable from any reasonable programming language. For example, Microsoft Internet Explorer itself is nothing but a tiny 89 KB program which wraps together dozens of very powerful components which are freely available to sophisticated Windows programmers and which are mostly designed to be flexible and powerful. Unfortunately, since programmers do not have access to the source code for those components, they can only be used in ways which were precisely foreseen and allowed for by the component developers at Microsoft, which doesn't always work out. And sometimes there are bugs, usually the fault of the person calling the API, which are difficult or impossible to debug without the source code. The Unix cultural value of visible source code makes it an easier environment to develop for. Any Windows developer will tell you about the time they spent four days tracking down a bug because, say, they thought that the memory size returned by LocalSize would be the same as the memory size they originally requested with LocalAlloc, or some similar bug they could have fixed in ten minutes if they could see the source code of the library. Raymond invents an amusing story to illustrate this which will ring true to anyone who has ever used a library in binary form.

So you get these religious arguments. Unix is better because you can debug into libraries. Windows is better because Aunt Marge gets some confirmation that her email was actually sent. Actually, one is not better than another, they simply have different values: in Unix making things better for other programmers is a core value and in Windows making things better for Aunt Marge is a core value.

Let's look at another cultural difference. Raymond says, "Classic Unix documentation is written to be telegraphic but complete... The style assumes an active reader, one who is able to deduce obvious unsaid consequences of what is said, and who has the self-confidence to trust those deductions. Read every word carefully, because you will seldom be told anything twice." Oy vey, I thought, he's actually teaching young programmers to write more impossible man pages.

For end users, you'll never get away with this. Raymond may call it "oversimplifying condescension," but the Windows culture understands that end users don't like reading and if they concede to read your documentation, they will only read the minimum amount, and so you have to explain things repeatedly... indeed the hallmark of a good Windows help file is that any single topic can be read by itself by an average reader without assuming knowledge of any other help topic.

How did we get different core values? This is another reason Raymond's book is so good: he goes deeply into the history and evolution of Unix and brings new programmers up to speed with all the accumulated history of the culture back to 1969. When Unix was created and when it formed its cultural values, there were no end users. Computers were expensive, CPU time was expensive, and learning about computers meant learning how to program. It's no wonder that the culture which emerged valued things which are useful to other programmers. By contrast, Windows was created with one goal only: to sell as many copies as conceivable at a profit. Scrillions of copies. "A computer on every desktop and in every home" was the explicit goal of the team which created Windows, set its agenda and determined its core values. Ease of use for non-programmers was the only way to get on every desk and in every home and thus usability über alles became the cultural norm. Programmers, as an audience, were an extreme afterthought.

The cultural schism is so sharp that Unix has never really made any inroads on the desktop. Aunt Marge can't really use Unix, and repeated efforts to make a pretty front end for Unix that Aunt Marge can use have failed, entirely because these efforts were done by programmers who were steeped in the Unix culture. For example, Unix has a value of separating policy from mechanism which, historically, came from the designers of X. This directly led to a schism in user interfaces; nobody has ever quite been able to agree on all the details of how the desktop UI should work, and they think this is OK, because their culture values this diversity, but for Aunt Marge it is very much not OK to have to use a different UI to cut and paste in one program than she uses in another. So here we are, 20 years after Unix developers started trying to paint a good user interface on their systems, and we're still at the point where the CEO of the biggest Linux vendor is telling people that home users should just use Windows. I have heard economists claim that Silicon Valley could never be recreated in, say, France, because the French culture puts such a high penalty on failure that entrepreneurs are not willing to risk it. Maybe the same thing is true of Linux: it may never be a desktop operating system because the culture values things which prevent it. OS X is the proof: Apple finally created Unix for Aunt Marge, but only because the engineers and managers at Apple were firmly of the end-user culture (which I've been imperialistically calling "the Windows Culture" even though historically it originated at Apple). They rejected the Unix culture's fundamental norm of programmer-centricity. They even renamed core directories -- heretical! -- to use common English words like "applications" and "library" instead of "bin" and "lib."

Raymond does attempt to compare and contrast Unix to other operating systems, and this is really the weakest part of an otherwise excellent book, because he really doesn't know what he's talking about. Whenever he opens his mouth about Windows he tends to show that his knowledge of Windows programming comes mostly from reading newspapers, not from actual Windows programming. That's OK; he's not a Windows programmer; we'll forgive that. As is typical from someone with a deep knowledge of one culture, he knows what his culture values but doesn't quite notice the distinction between parts of his culture which are universal (killing old ladies, programs which crash: always bad) and parts of the culture which only apply when you're programming for programmers (eating raw fish, command line arguments: depends on audience).

There are too many monocultural programmers who, like the typical American kid who never left St. Paul, Minnesota, can't quite tell the difference between a cultural value and a core human value. I've encountered too many Unix programmers who sneer at Windows programming, thinking that Windows is heathen and stupid. Raymond all too frequently falls into the trap of disparaging the values of other cultures without considering where they came from. It's rather rare to find such bigotry among Windows programmers, who are, on the whole, solution-oriented and non-ideological. At the very least, Windows programmers will concede the faults of their culture and say pragmatically, "Look, if you want to sell a word processor to a lot of people, it has to run on their computers, and if that means we use the Evil Registry instead of elegant ~/.rc files to store our settings, so be it." The very fact that the Unix world is so full of self-righteous cultural superiority, "advocacy," and slashdot-karma-whoring sectarianism while the Windows world is more practical ("yeah, whatever, I just need to make a living here") stems from a culture that feels itself under siege, unable to break out of the server closet and hobbyist market and onto the mainstream desktop. This haughtiness-from-a-position-of-weakness is the biggest flaw of The Art of UNIX Programming, but it's not really a big flaw: on the whole, the book is so full of incredibly interesting insight into so many aspects of programming that I'm willing to hold my nose during the rare smelly ideological rants because there's so much to learn about universal ideals from the rest of the book. Indeed I would recommend this book to developers of any culture in any platform with any goals, because so many of the values which it trumpets are universal. When Raymond points out that the CSV format is inferior to the /etc/passwd format, he's trying to score points for Unix against Windows, but, you know what? He's right. /etc/passwd is easier to parse than CSV, and if you read this book, you'll know why, and you'll be a better programmer.


Big Macs vs. The Naked Chef

Thursday, January 18, 2001

Mystery: why is it that some of the biggest IT consulting companies in the world do the worst work?

Why is it that the cool upstart consulting companies start out with a string of spectacular successes, meteoric growth, and rapidly degenerate into mediocrity?

I've been thinking about this, and thinking about how Fog Creek Software (my own company) should grow. And the best lessons I can find come from McDonald's. Yes, I mean the awful hamburger chain.

The secret of Big Macs is that they're not very good, but every one is not very good in exactly the same way. If you're willing to live with not-very-goodness, you can have a Big Mac with absolutely no chance of being surprised in the slightest.

The other secret of Big Macs is that you can have an IQ that hovers somewhere between "idiot" and "moron" (to use the technical terms) and you'll still be able to produce Big Macs that are exactly as unsurprising as all the other Big Macs in the world. That's because McDonald's real secret sauce is its huge operations manual, describing in stunning detail the exact procedure that every franchisee must follow in creating a Big Mac. If a Big Mac hamburger is fried for 37 seconds in Anchorage, Alaska, it will be fried for 37 seconds in Singapore - not 36, not 38. To make a Big Mac you just follow the damn rules.

The rules have been carefully designed by reasonably intelligent people (back at McDonald's Hamburger University) so that dumdums can follow them just as well as smart people. In fact the rules include all kinds of failsafes, like bells that go off if you keep the fries in the oil too long, which were created to compensate for more than a little human frailty. There are stopwatches and timing systems everywhere. There is a system to make sure that the janitor checks if the bathrooms are clean every half hour. (Hint: they're not.)

The system basically assumes that everybody will make a bunch of mistakes, but the burgers that come out will be, um, consistent, and you'll always be asked if you want fries with that.

Just for the sake of amusement, let's compare a McDonald's cook, who is following a set of rules exactly and doesn't know anything about food, to a genius like The Naked Chef, the British cutie Jamie Oliver. (If you chose to leave this site now and follow that link to watch the MTV-like videos of The Naked Chef making basil aioli, you have my blessing. Go in good health.) Anyway, comparing McDonald's to a gourmet chef is completely absurd, but please suspend disbelief for a moment, because there's something to be learned here.

Now, the Naked Chef doesn't follow no stinkin' Operations Manual. He doesn't measure anything. While he's cooking, you see a flurry of food tossed around willy-nilly. "We'll just put a bit of extra rosemary in there, that won't hurt, and give it a good old shake," he says. " Mash it up. Perfect. Just chuck it all over the place." (Yes, it really looks like he's just chucking it all over the place. Sorry, but if I tried to chuck it all over the place, it wouldn't work.) It takes about 14 seconds and he's basically improvised a complete gourmet meal with roasted slashed fillet of sea-bass stuffed with herbs, baked on mushroom potatoes with a salsa-verde. Yum.

Well, I think it's pretty obvious that The Naked Chef's food is better than you get at McDonald's. Even if it sounds like a stupid question, it's worth a minute to ask why. It's not such a stupid question. Why can't a big company with zillions of resources, incredible scale, access to the best food designers money can buy, and infinite cash flow produce a nice meal?

Imagine that The Naked Chef gets bored doing "telly" and opens a restaurant. Of course, he's a brilliant chef, the food would be incredible, so the place is hopping with customers and shockingly profitable.

When you have a shockingly profitable restaurant, you quickly realize that even if you fill up every night, and even if you charge $19 for an appetizer and $3.95 for a coke, your profits reach a natural limit, because one chef can only make so much food. So you hire another chef, and maybe open some more branches, maybe in other cities.

Now a problem starts to develop: what we in the technical fields call the scalability problem. When you try to clone a restaurant, you have to decide between hiring another great chef of your caliber (in which case, that chef will probably want and expect to keep most of the extra profits that he created, so why bother), or else you'll hire a cheaper, younger chef who's not quite as good, but pretty soon your patrons will figure that out and they won't go to the clone restaurant.

The common way of dealing with the scalability problem is to hire cheap chefs who don't know anything, and give them such precise rules about how to create every dish that they "can't" screw it up. Just follow these here rules, and you'll make great gourmet food!

Problem: it doesn't work exactly right. There are a million things that a good chef does that have to do with improvisation. A good chef sees some awesome mangos in the farmer's market and improvises a mango-cilantro salsa for the fish of the day. A good chef deals with a temporary shortage of potatoes by creating some taro chip thing. An automaton chef who is merely following instructions might be able to produce a given dish when everything is working perfectly, but without real talent and skill, will not be able to improvise, which is why you never see jicama at McDonald's.

McDonald's requires a very particular variety of potato, which they grow all over the world, and which they pre-cut and freeze in massive quantities to survive shortages. The precutting and freezing means that the french-fries are not as good as they could be, but they are certainly consistent and require no chef-skills. In fact, McDonald's does hundreds of things to make sure that their product can be produced with consistent quality, by any moron you can get in the kitchen, even if the quality is "a bit" lower.

Summary, so far:

  1. Some things need talent to do really well.
  2. It's hard to scale talent.
  3. One way people try to scale talent is by having the talent create rules for the untalented to follow.
  4. The quality of the resulting product is very low.

You can see the exact same story playing out in IT consulting. How many times have you heard this story?

Mike was unhappy. He had hired a huge company of IT consultants to build The System. The IT consultants he hired were incompetents who kept talking about "The Methodology" and who spent millions of dollars and had failed to produce a single thing.

Luckily, Mike found a youthful programmer who was really smart and talented. The youthful programmer built his whole system in one day for $20 and pizza. Mike was overjoyed. He recommended the youthful programmer to all his friends.

Youthful Programmer starts raking in the money. Soon, he has more work than he can handle, so he hires a bunch of people to help him. The good people want too many stock options, so he decides to hire even younger programmers right out of college and "train them" with a 6 week course.

The trouble is that the "training" doesn't really produce consistent results, so Youthful Programmer starts creating rules and procedures that are meant to make more consistent results. Over the years, the rule book grows and grows. Soon it's a six-volume manual called The Methodology.

After a few dozen years, Youthful Programmer is now a Huge Incompetent IT Consultant with a capital-M-methodology and a lot of people who blindly obey the Methodology, even when it doesn't seem to be working, because they have no bloody idea whatsoever what else to do, and they're not really talented programmers -- they're just well-meaning Poli Sci majors who attended the six-week course.

And Newly Huge Incompetent IT Consultant starts messing up. Their customers are unhappy. And another upstart talented programmer comes and takes away all their business, and the cycle begins anew.

I don't need to name names, here, this cycle has happened a dozen times. All the IT service companies get greedy and try to grow faster than they can find talented people, and they grow layers upon layers of rules and procedures which help produce "consistent," if not very brilliant work.

But the rules and procedures only work when nothing goes wrong. Various "data-backed Web site" consulting companies sprouted up in the last couple of years and filled their ranks by teaching rank amateurs the fourteen things you need to know to create a data-backed Web site ("here's a select statement, kid, build a Web site"). But now that dotcoms are imploding and there's suddenly demand for high-end GUI programming, C++ skills, and real computer science, the kids who only have select statements in their arsenal just have too steep a learning curve and can't catch up. But they still keep trying, following the rules in chapter 17 about normalizing databases, which mysteriously don't apply to The New World. The brilliant founders of these companies could certainly adjust to the new world: they are talented computer scientists who can learn anything, but the company they built can't adjust because it has substituted a rulebook for talent, and rulebooks don't adjust to new times.

What's the moral of the story? Beware of Methodologies. They are a great way to bring everyone up to a dismal, but passable, level of performance, but at the same time, they are aggravating to more talented people who chafe at the restrictions that are placed on them. It's pretty obvious to me that a talented chef is not going to be happy making burgers at McDonald's, precisely because of McDonald's rules. So why do IT consultants brag so much about their methodologies? (Beats me.)

What does this mean for Fog Creek? Well, our goal has never been to become a huge consulting company. We started out doing consulting as a means to an end -- the long-term goal was to be a software company that is always profitable, and we achieved that by doing some consulting work to supplement our software income. After a couple of years in business our software revenues grew to the point where consulting was just a low-margin distraction, so now we only do consulting engagements that directly support our software. Software, as you know, scales incredibly well. When one new person buys FogBUGZ, we make more money without spending any more money.

More important is our obsession with hiring the best... we are perfectly happy to stay small if we can't find enough good people (although with six weeks annual vacation, finding people doesn't seem to pose a problem). And we refuse to grow until the people we already hired have learned enough to become teachers and mentors of the new crowd.


Bionic Office

Wednesday, September 24, 2003

Well.

That took rather longer than expected.

We have,

Most software managers know what good office space would be like, and they know they don't have it, and can't have it. Office space seems to be the one thing that nobody can get right and nobody can do anything about. There's a ten year lease, and whenever the company moves the last person anybody asks about how to design the space is the manager of the software team, who finds out what his new veal-fattening pens, uh, cubicle farm is going to be like for the first time on the Monday after the move-in.

Well, it's my own damn company and I can do something about it, so I did.

Maybe I'm just an architecture queen. I probably pay more attention to my physical surroundings than the average software developer. I might take it too seriously. But there are three reasons I take it so seriously:

  • There's a lot of evidence that the right kind of office space can improve programmer productivity, especially private offices.
  • Having drop-dead gorgeous, private, windowed offices makes it a lot easier to recruit the kinds of superstars that produce ten times as much as the merely brilliant software developers. If I have to compete at New York salaries against Bangalore salaries, I'm going to need those superstars, so when people come in for an interview, I need to see jaws on the floor. It's about drama.
  • Hey, this is my job; this is where I spend my days; it's my time away from my friends and family. It better be nice.
Working with architect

Architects use the term "brief" for what we software developers call "system requirements." Here was the brief I gave Roy.

  1. Private offices with doors that close were absolutely required and not open to negotiation.
  2. Programmers need lots of power outlets. They should be able to plug new gizmos in at desk height without crawling on the floor.
  3. We need to be able to rewire any data lines (phone, LAN, cable TV, alarms, etc.) easily without opening any walls, ever.
  4. It should be possible to do pair programming.
  5. When you're working with a monitor all day, you need to rest your eyes by looking at something far away, so monitors should not be up against walls.
  6. The office should be a hang out: a pleasant place to spend time. If you're meeting your friends for dinner after work you should want to meet at the office. As Philip Greenspun bluntly puts it: "Your business success will depend on the extent to which programmers essentially live at your office. For this to be a common choice, your office had better be nicer than the average programmer's home. There are two ways to achieve this result. One is to hire programmers who live in extremely shabby apartments. The other is to create a nice office."

Roy did a great job. This is what you pay an architect for. I predict he will become something of a world expert on designing offices for software teams. Here's how he translated my brief into three dimensional space.

Private Offices. Not only did we get spacious, windowed private offices, but even the common area workstations (for non-developers) are hidden in clever angular alcoves, so everyone gets their own private space without line of sight to anyone else.

The walls between the offices and the workstations are made of high tech, translucent acrylic which glows softly and provides natural light to the interior without reducing privacy.

Power. Every desk has twenty, that's right, twenty outlets. Four of them are colored orange and have uninterruptible power coming off of a UPS in the server closet, so you don't need a UPS in every office.

The outlets are right below desk level in a special trough which runs the entire length of the desk, about six inches deep and six inches wide. The trough is a place to hide all your cables neatly and has a handy cover which blends in with the desk.

Wiring. There is a Snake Tray system running near the ceiling from the server room and throughout the office, running through every room. It is completely accessible so if you want to run any kind of (low voltage) cable from point A to point B you can do this neatly. We only moved in Friday, and we've already redone the intra-office LAN wiring, in a project which took about half an hour, so the snake tray has already proven itself. Every office has its own 8-port network switch, so you can plug in your laptop and your desktop and your Macintosh and that old computer you keep around to read Joel on Software when your main computer is rebooting to install today's Windows Update, and still have 3 ports left over (attention math geniuses: no need to email. One port is the uplink.) I sneer at silly building managers who still think that one LAN port per office is about right. For lawyers, maybe.

Pair Programming. When you make typical L-shaped desks many developers set themselves up in the corner. When they need to collaborate temporarily, or pair program, or even just show something to someone on their screen, the second person has to either lean all the way across the desk or look over the first person's shoulder. To avoid this we designed all the desks to be long and straight so that wherever a software developer sits, there's always room for another person to pull up a chair and sit next to them.

Resting eyes. Although the desks are up against the walls, there is an interior window in that wall, which cleverly looks across the corner of the next developer's office and through his window. Because of the rather brilliant layout this doesn't reduce privacy because even though you have a window onto the next office, it is angled so that from most positions you really only look across a small corner of that room and out its exterior window. The net result is that every office has windows on three sides, two of which look outside, creating the architectural pattern Light on Two Sides of Every Room. This is quite an accomplishment: you try coming up with a scheme to give everyone a corner office in a conventional building. Another reason hiring a great architect was well worth the money.

Hang out. We furnished the office with a kitchenette and a lounge area with sofas and a huge HDTV plasma screen with DVD player. We're planning a pool table and game console. Private offices means you can listen to music at reasonable volumes without headphones and nobody will care.

Bottom Line it For Me.

The monthly rent for our offices, when fully occupied, will run about $700 per employee. The build-out was done on budget and paid for almost entirely by the landlord. I suspect that $700 per person is on the high side for software developers throughout the world, but if it means we can hire from the 99.9 percentile instead of the 99 percentile, it'll be worth it.


Building Communities with Software

Monday, March 03, 2003

The social scientist Ray Oldenburg talks about how humans need a third place, besides work and home, to meet with friends, have a beer, discuss the events of the day, and enjoy some human interaction. Coffee shops, bars, hair salons, beer gardens, pool halls, clubs, and other hangouts are as vital as factories, schools and apartments ["The Great Good Place", 1989]. But capitalist society has been eroding those third places, and society is left impoverished. In "Bowling Alone," Robert Putnam brings forth, in riveting and well-documented detail, reams of evidence that American society has all but lost its third places. Over the last 25 years, Americans "belong to fewer organizations that meet, know our neighbors less, meet with friends less frequently, and even socialize with our families less often." [2000] For too many people, life consists of going to work, then going home and watching TV. Work-TV-Sleep-Work-TV-Sleep. It seems to me that the phenomenon is far more acute among software developers, especially in places like Silicon Valley and the suburbs of Seattle. People graduate from college, move across country to a new place where they don't know anyone, and end up working 12 hour days basically out of loneliness.

So it's no surprise that so many programmers, desperate for a little human contact, flock to online communities - chat rooms, discussion forums, open source projects, and Ultima Online. In creating community software, we are, to some extent, trying to create a third place. And like any other architecture project, the design decisions we make are crucial. Make a bar too loud, and people won't be able to have conversations. That makes for a very different kind of place than a coffee shop. Make a coffee shop without very many chairs, as Starbucks does, and people will carry their coffee back to their lonely rooms, instead of staying around and socializing like they do in the fantasy TV coffeehouse of "Friends," a program we watch because an ersatz third place is less painful than none at all.

In software, as in architecture, design decisions are just as important to the type of community that develops or fails to develop. When you make something easy, people do it more often. When you make something hard, people do it less often. In this way you can gently encourage people to behave in certain ways which determine the character and quality of the community. Will it feel friendly? Is there thick conversation, a European salon full of intellectuals with interesting ideas? Or is the place deserted, with a few dirty advertising leaflets lying around on the floor that nobody has bothered to pick up?

Look at a few online communities and you'll instantly notice the different social atmosphere. Look more closely, and you'll see this variation is most often a byproduct of software design decisions.

On Usenet, threads last for months and months and go off onto so many tangents that you never know where they've been. Whenever a newbie stumbles by and asks a germane question, the old timers shout him down and tell him to read the FAQ. Quoting, with the ">" symbol, is a disease that makes it impossible to read any single thread without boring yourself to death by re-reading the whole history of a chain of argument which you just read in the original, seconds ago, again and again and again. Shlemiel the Painter reading.

On IRC, you can't own your nickname and you can't own a channel -- once the last person leaves a room, anyone can take it over. That's the way the software works. The social result was that it was often impossible to find your friends when you came back the next day, because someone else might have locked you out of your chatroom and your friends might have been forced to choose different nicknames. The only way to prevent gay bashers in Perth, Australia from taking over gay chat channels when the boys went to sleep was to create a software robot to hang around 24 hours a day and guard the channel. Many IRC participants put more effort into complicated bot-wars, attempts to take over channels, and general tomfoolery than actually having a conversation, often ruining things for the rest of us.

On most investment discussion boards, it's practically impossible to follow a thread from beginning to end, because every post is its own page, which makes for a lot of banner ad inventory, but the latency in reading a conversation will eventually drive you nuts. The huge amount of flashing commercial crap on all four sides of the conversation makes you feel like you were trying to make friends in Times Square, but the neon lights keep demanding all the attention.

On Slashdot, every thread has hundreds of replies, many of which are identical, so the conversation there feels insipid and stupid. In a moment I'll reveal why Slashdot has so many identical replies and the Joel on Software forum doesn't.

And on FuckedCompany.com, the discussion board is completely, utterly worthless; the vast majority of posts consist of irrelevant profanity and general abusiveness and it feels like a fraternity rudeness contest, without any fraternity.

So, we have discovered the primary axiom of online communities:

Small software implementation details result in big differences in the way the community develops, behaves, and feels.

IRC users organize themselves around bot warfare because the software doesn't let you reserve a channel. Usenet threads are massively redundant because the original Usenet reader, "rn," designed for 300 baud modems, never shows you old posts, only new ones, so if you want to nitpick about something someone said, you had to quote them or your nitpick won't make sense.

With that in mind, I'd like to answer the most common questions about the Joel on Software forum, why it was designed the way it was designed, how that makes it work, and how it could be improved.

Q. Why is the software so dang simplistic?

A. In the early days of the Joel on Software forum, achieving a critical mass to get the conversation off the ground was important to prevent the empty restaurant phenomenon (nobody goes into an empty restaurant, they'll always go into the full one next door even if it's totally rubbish.) Thus a design goal was to eliminate impediments to posting. That's why there's no registration and there are literally no features, so there's nothing to learn.

The business goal of the software that runs the forum was to provide tech support for Fog Creek's products. That's what paid for the development. To achieve that goal, nothing was more important than making the software super simple so that anyone could be comfortable using it. Everything about how the forum works is incredibly obvious. I don't know of anyone who hasn't been able to figure out how to use it immediately.

Q. Could you make a feature where I check a box that says "email me if somebody replies to my post?"

A. This one feature, so easy to implement and thus so tempting to programmers, is the best way to kill dead any young forum. Implement this feature and you may never get to critical mass. Philip Greenspun's LUSENET has this feature and you can watch it sapping the life out of young discussion groups.

Why?

What happens is that people go to the group to ask a question. If you offer the "notify me" checkbox, these people will post their question, check the box, and never come back. They'll just read the replies in their mailbox. The end.

If you eliminate the checkbox, people are left with no choice but to check back every once in a while. And while they're checking back, they might read another post which looks interesting. And they might have something to contribute to that post. And in the critical early days when you're trying to get the discussion group to take off, you've increased the "stickiness" and you've got more people hanging around, which helps achieve critical mass a lot quicker.

Q. OK, but can't you at least have branching? If someone gets off on a tangent, that should be its own branch which you can follow or go back to the main branch.

A. Branching is very logical to a programmer's mind but it doesn't correspond to the way conversations take place in the real world. Branched discussions are disjointed to follow and distracting. You know what I find distracting? When I'm trying to do something on my bank's web site and the site is so slow I can't remember what I'm doing from one click to the next. That reminds me of a joke. Three old ladies talking. Lady 1: "I'm so forgetful the other day I was on the steps to my apartment with a bag, and I couldn't remember if I was taking out the trash or going upstairs with the groceries." Lady 2: "I'm so forgetful I was in my car in the driveway and I couldn't remember if I was coming home or going to shul." Lady 3: "Thank God, I still have my memory, clear as a bell, knock on wood. (knock knock knock). Come in, door's open!" Branching makes discussions get off track, and reading a thread that is branched is discombobulating and unnatural. Better to force people to start a new topic if they want to get off topic. Which reminds me...

Q. Your list of topics is sorted wrong. It should put the topic with the most recent reply first, rather than listing them based on the time of the original post.

A. It could do that; that's what many web-based forums do. But when you do that certain topics tend to float near the top forever, because people will be willing to argue about H1B visas, or what's wrong with Computer Science in college, until the end of the universe. Every day 100 new people arrive in the forum for the first time, and they start at the top of the list, and they dive into that topic with gusto.

The way I do it has two advantages. One, topics rapidly go away, so conversation remains relatively interesting. Eventually people have to just stop arguing about a given point.

Two, the order of topics on the home page is stable, so it's easier to find a topic again that you were interested in because it stays in the same place relative to its neighbors.

Q. Why don't you have some kind of system so I can see what posts I've already read?

A. We have the best system that can be implemented in a distributed, scalable fashion: we let everyone's browser keep track of it. Web browsers will change the color of the links you've already visited from blue to purple. So all we have to do is subtly change the URL for each topic to include the number of replies available; that way when there are additional replies the post will appear in the "unread" color again.

Anything more elaborate than this would be harder to build and would needlessly complicate the UI.

Q. The damn "reply" link is all the way at the bottom. This is a usability annoyance because you have to scroll all the way to the bottom.

A. This is intentional. I would prefer that you read all the posts before you reply, otherwise you may post something which is repetitive or which sounds disjointed coming after the previous last post. Of course, I can't physically grab your eyeballs and move them from left to right, forcing you to read the entire thread before letting you post, but if I put a "reply" link anywhere but the bottom of the page that would positively encourage people to spew their little gems before they've read what's already there. This is why Slashdot topics have 500 replies but only 17 interesting replies, and it's why nobody likes to read Slashdot discussions: they sound like a classroom full of children all shouting out the same answer at the same time. ("ha ha ... Bill Gates! That's an oxymoron!")

Q. The damn "Start a New Topic" link is all the way at the bottom...

A. Uh huh, same thing.

Q. Why don't you show people their posts to confirm them before you post them? Then people wouldn't make mistakes and typos.

A. Empirically, that is not true. Not only is it not true, it's the opposite of true.

Part one: when you have a confirmation step, most people just click past it. Very few people reread their post carefully. If they wanted to reread their post carefully, they could have done it while they were editing it, but they are bored by their post already, it's yesterday's newspaper, they are ready to move on.

Part two: the lack of the confirmation step actually makes people more cautious. It's like those studies they did that showed that it's safer, on twisty mountain roads, to remove the crash barrier, because it makes people scared and so they drive more carefully, and any way, that little flimsy aluminum crash barrier ain't gonna stop a 2 ton SUV moving at 50 mph from flying off the cliff. You're better off, statistically, just scaring the bejesus out of drivers so they creep along at 2 miles per hour around the hairpins.

Q. Why don't you show me the post I'm replying to, while I compose my reply?

A. Because that will tempt you to quote a part of it in your own reply. Anything I can do to reduce the amount of quoting will increase the fluidity of the conversation, making topics interesting to read. Whenever someone quotes something from above, the person who reads the topic has to read the same thing twice in a row, which is pointless and automatically guaranteed to be boring.

Sometimes people still try to quote things, usually because they are replying to something from three posts ago, or because they're mindlessly nitpicking and they need to rebut 12 separate points. These are not bad people, they're just programmers, and programming requires you to dot every i and cross every t, so you get into a frame of mind where you can't leave any argument unanswered any more than you would ignore an error from your compiler. But I'll be damned if I make it EASY on you. I'm almost tempted to try to find a way to show posts as images so you can't cut and paste them. If you really need to reply to something from three posts ago, kindly take a moment to compose a decent English sentence ("When Fred said blah, he must not have considered..."), don't litter the place with your <<<>>>s.

Q. Why do posts disappear sometimes?

A. The forum is moderated. That means that a few people have the magick powah to delete a post. If the post they delete is the first one in a thread, the thread itself appears deleted because there's no way to get to it.

Q. But that's censorship!

A. No, it's picking up the garbage in the park. If we didn't do it, the signal to noise ratio would change dramatically for the worse. People post spam and get rich schemes, people post antisemitic comments about me, people post nonsense that doesn't make any sense. Some idealistic youngsters may imagine a totally uncensored world as one in which the free exchange of intelligent ideas raises everyone's IQ, an idealized Oxford Debate Society or Speakers' Corner. I am pragmatic and understand that a totally uncensored world just looks like your inbox: 80% spam, advertising, and fraud, rapidly driving away the few interesting people.

If you are looking for a place to express yourself in which there will be no moderation, my advice to you would be to (a) create a new forum and (b) make it popular. [Apologies to Larry Wall].

Q. How do you decide what to delete?

A. First of all, I remove radically off-topic posts or posts which, in my opinion, are only of interest to a very small number of people. If something is not about the same general topics as Joel on Software is about, it may be interesting as all heck to certain people but it's not likely to interest the majority of people who came to my site to hear about software development.

My policy in the past has been that "off topic" includes any discussion of the forum itself, its design or usability. There's a slightly different reason for this, almost another axiom. Every forum, mailing list, discussion group, and BBS will, all else being equal, lapse into conversations about the forum itself every week or two. Literally once a week somebody strolls in and announces his list of improvements to the forum software which he demands be made right away. And then somebody says, "look buddy you're not paying for it Joel's doing us a favor get lost." And somebody else says "Joel's not doing this out of the goodness of his heart it's marketing for Fog Creek." And it's just SOOOO BORING because it happens EVERY WEEK. It's like talking about the weather when you have nothing else to talk about. It may be exciting to the new person who just appeared on the board but it is only barely about software development, so, as Strong Bad says, "DELETED". Unfortunately what I have learned is that trying to get people to stop talking about the forum is like trying to stop a river. But please, if you're reading this article and you want to discuss it on the forum, please, please, do me a huge favor, and resist the urge.

We will delete posts which are personal, ad hominem attacks on non public personalities. I better define that. Ad hominem means it is an attack on the individual, rather than on his ideas. If you say "that is a stupid idea because..." it's OK. If you say "you are stupid" then it's an ad hominem attack. If it's vicious or uncivil or libelous, I delete it. There's one exception: because the Joel on Software forum is the best place to criticize Joel, vicious or uncivil posts about Joel are allowed to stand but only if they contain some tiny sliver of a useful argument or idea.

I automatically delete posts which comment on the spelling or grammar of a previous poster. We'll be talking about interviews and someone will say, "It's a wonder you can get a job with spelling like that." It's just super boring to talk about other people's spelling. SUPER, SUPER boring.

Q. Why don't you just post the rules instead of leaving it as a mystery?

A. The other day I was taking the train from the Newark Airport back to Manhattan. Besides being in general disrepair, the only thing to read was a large sign that explained very sternly and in great detail that if you misbehaved, you would be put off the train at the next stop and the police would be summoned. And I thought, 99.99999% of the people who read that sign ain't gonna be misbehavin', and the misbehavors couldn't care less what the sign says. So the net result of the sign is to make honest citizens feel like they're being accused of something, and it doesn't deter the sociopaths at all, and it just reminds the good citizens of New Jersey endlessly that they're in Newark, Crime Capital, where sociopaths get on the train and do Unpleasant Things and Make a Scene and have to be Put Off and the Police Summoned.

Almost everyone on the Joel on Software forum, somehow, was born with the part of the brain that tells them that it's not civilized to post vicious personal attacks, or to post questions about learning French on a software forum, or to conduct an argument by criticizing someone's spelling. And the other .01% don't care about the rules. So posting rules is just a way to insult the majority of the law-abiding citizens and it doesn't deter the morons who think their own poo smells delicious and nothing they post could possibly be against the rules.

When you address troublemakers in public, everyone else thinks you're paranoid or feels angry at being scolded when they did nothing wrong. It's like being in grade school again, and one idiot-child has broken a window, and now everyone has to sit there listening to the teacher giving the whole class a stern lecture on why you mustn't break windows. So any public discussion of why a particular post got deleted, for example, is taboo.

Q. Instead of deleting posts, why don't you have a moderation scheme, where people vote on how much they like a post, and people can choose how high the vote has to be before they read it?

A. This is, of course, how Slashdot works, and I'll bet you 50% of the people who read Slashdot regularly have never figured it out.

There are three things I don't like about this. One: it's more UI complication, a feature that people need to learn how to use. Two: it creates such complicated politics that it make the Byzantine Empire look like 3rd grade school government. And three: when you read Slashdot with the filters turned up high enough that you only see the interesting posts, the narrative is completely lost. You just get a bunch of random disjointed statements with no context.

Q. Why don't you have a registration scheme to eliminate rude posters?

A. As I explained earlier, the goal of the forum is to make it easy to post. (Remember, the software was written for tech support.) Registration schemes eliminate at least 90% of the people who might have posted, and in a tech support scenario, those 90% are going to call my toll free number.

Besides, I don't think registration would help. If somebody is being abusive, it doesn't help to ban them, they can trivially reregister. The idea of improving the community by requiring registration is an old one, and it's appropriate, I think, for the Echo/Well type of conferences where you're creating a network of people as much as you're discussing a topic, and you charge people cash money to belong.

But requiring registration does NOT improve the quality of the conversation or the average quality of the participants. If you look closely at the signal-to-noise ratio on the Joel on Software forum, you might start to notice that the noisiest people (i.e. the people who post the most words while contributing the fewest ideas) are often the long time, hard core members who visit the forum every ten minutes. These are the people who feel the need to chime in with a "I agree with that" and replies to Every Single Topic even when they haven't got an original thought to contribute. And they would certainly register.

Q. Any plans for the future?

A. Working on the software for the discussion forum is not a priority for me or my company: it's good enough, it works, it has created an interesting place to talk about hard computer management problems and get ideas from some of the smartest people in the world. And I've got too many better things to work on. Somebody else can create the next big leap in usability for discussion forums.

I just created a New York City forum, to see if geographically based forums encourage people to get to know each other in person as well as online. In my experience, regionally- based communities cause the community to take a giant leap from a simple website to a real society, a true third place.

Creating community, in any case, is a noble goal, because it's sorely missing for so many of us. Let's keep plugging away at it.


Camels and Rubber Duckies

Wednesday, December 15, 2004

You've just released your latest photo-organizing software. Through some mechanism which will be left as an exercise to the reader, you've managed to actually let people know about it. Maybe you have a popular blog or something. Maybe Walt Mossberg wrote a rave review in the Wall Street Journal.

One of the biggest questions you're going to be asking now is, "How much should I charge for my software?" When you ask the experts they don't seem to know. Pricing is a deep, dark mystery, they tell you. The biggest mistake software companies make is charging too little, so they don't get enough income, and they have to go out of business. An even bigger mistake, yes, even bigger than the biggest mistake, is charging too much, so they don't get enough customers, and they have to go out of business. Going out of business is not good because everybody loses their job, and you have to go work at Wal*Mart as a greeter, earning minimum wage and being forced to wear a polyester uniform all day long.

So if you like cotton uniforms you better get this right.

The answer is really complicated. I'm going to start with a little economic theory, then I'm going to tear the theory to bits, and when I'm finished, you'll know a lot more about pricing and you still won't know how much to charge for your software, but that's just the nature of pricing. If you can't be bothered to read this, just charge $0.05 for your software, unless it does bug tracking, in which case charge $30,000,000 for it.

Now. Where was I.

Some Economic Theory

Imagine, for the moment, that your software costs $199. Why $199? Well, because I have to start somewhere. We'll try other numbers real soon now. For now, imagine that you charge $199 and that gets you 250 customers.

Let me plot that:

This little chart I made means that if you charge $199, 250 people will buy your software. (As you see, economists are very odd people, and they like to put the quantity sold on the x-axis with the price on the y-axis. If 250 people bought your software, it must mean that you charged $199!)

What would happen if you raised the price to $249?

Some of the people who might have been willing to pay $199 are going to think $249 is too much, so they'll drop out.

Obviously, people who wouldn't even buy it for $199 are certainly not going to buy it at the higher price.

If 250 people bought at $199, we must assume that less than 250 people would buy it at $249. Let's guess, oh, 200:

What if we charged less? Say, $149? Well, everyone who would buy it for $199 will certainly buy it for $149, and there will probably be even more people who think $149 is affordable, so let's say we could sell 325 copies at $149:

And so on and so forth:

In fact rather than graphing a few discrete points here, let's draw the complete curve that includes all these points, and while I'm at it I'll fix the x-axis so it's to scale:

Now you can tell me any price between $50 and $400 and I'll tell you how many people will buy your software at that price. What we have here is a classic demand curve and the demand curve is always downward-sloping, because the more you charge, the fewer people will be willing to buy your software.

These numbers are not, of course, real. The only thing I'm asking you to believe, so far, is that the demand curve is downward sloping.

(If it's still bothering you that I put the quantity on the x-axis and the price on the y-axis, when clearly the quantity is a function of the price, not the other way around, please take it up with Augustin Cournot. He probably has a blog by now.)

So how much should you charge?

"Uh, $50, because then I sell the most units!"

No no no. You're not trying to maximize units, you're trying to maximize profits.

Let's calculate profits.

Assume each unit of software that you sell costs you an incremental $35.

Maybe it cost you $250,000 to develop the software in the first place, but that's a sunk cost. We don't care about that anymore, because the $250,000 is the same whether you sell 1000 units or 0. Sunk. Kiss it goodbye. Set any price you want, the $250,000 is gone and therefore not relevant any more.

At this point all we can worry about is the incremental cost of selling each additional unit. That might include shipping and handling, that might include tech support, bank service charges, CD duplication and shrinkwrapping, whatever; I'm going to be really hand-wavy here and use $35 as my incremental cost.

Now we can whip out our handy-dandy copy of Visicalc:

Here's how to read that spreadsheet. Each row is a scenario. Row 3: If we were to charge $399, then we would sell 12 copies, making $364 profit per copy, for a total profit of $4368.

NOW WE'RE GETTING SOMEWHERE!

This is really cool. I think we're on the verge of solving the problem of how much to charge for software! I'M SO EXCITED!

The reason I'm so excited is it looks like if you plot price against profit, you get a nice curve with a big hump in the middle! And we all know what humps mean! Humps mean local maxima! Or camels. But here they mean local maxima!

In this chart the actual data is shown as little blue diamonds, and I've asked Excel to draw a nice polynomial trendline on top. So now all I have to do is drop a line straight down from the peak of the hump to find out the price I should charge to make a maximal amount of profit:

"O frabjous day! Callooh! Callay!" I chortle. We have found the optimum price, $220, and that's how much you should charge for your software. Thanks for your time.

Ahem.

Thank you for your time! Nothing more to see here! Move along now!

You're not leaving.

I see.

Some of the more observant members of my audience have detected through careful analysis of the scrollbar position in their web browser that I might have something more to say other than "$220."

Well, maybe. There's just a tiny little loose end I left untied which I might as well tie up now if you're all still up for it. Ok? OK!

You see, by setting the price at $220, we managed to sell, let's say, 233 copies of the software, at a total profit of $43,105, which is all good and fine, but something is distracting me: all those people who were all ready to pay more, like those 12 fine souls who would have paid a full $399, and yet, we're only charging them $220 just like everyone else!

The difference between $399 and $220, i.e., $179, is called consumer surplus. It's the extra value that those rich consumers got from their purchase that they would have been perfectly happy to do without.

It's sort of like if you were all set to buy that new merino wool sweater, and you thought it was going to cost $70, which is well worth it, and when you got to Banana Republic it was on sale for only $50! Now you have an extra $20 in found money that you would have been perfectly happy to give to the Banana Republicans!

Yipes!

That bothers good capitalists. Gosh darn it, if you're willing to do without it, well, give it to me! I can put it to good use, buying a SUV or condo or Mooney or yacht one of those other things capitalists buy!

In economist jargon, capitalists want to capture the consumer surplus.

Let's do this. Instead of charging $220, let's ask each of our customers if they are rich or if they are poor. If they say they're rich, we'll charge them $349. If they say they're poor, we'll charge them $220.

Now how much do we make? Back to Excel:

Notice the quantities: we're still selling the same 233 copies, but the richest 42 customers, who were all willing to spend $349 or more, are being asked to spend $349. And our profits just went up! from $43K to about $48K! NICE!

Capture me some more of that consumer surplus stuff!

But wait, that's not all. After selling all these 233 copies, I felt kind of bad about the people who were only willing to spend $99. After all, if I could sell a few more copies to those guys at $99, I'd still be making some money, since my marginal cost is only $35.

What if we called up all the customers who said "no thanks" at $220, and offered them the software at $99?

At $99, we have 450 potential customers, but don't forget that 233 of them already paid full price, leaving us with 217 extra customers who just wouldn't go along with the program and pay the full price:

Babymosesinabasket, I think we just made $62K in profit! All in all, an extra twenty thousand buckeroos, cash spendable money, which goes a loooong way towards a down payment on that fishing boat you've had your eye on. All from the power of segmentation: separating your customers into different groups according to how much they are willing to pay, and extracting the maximal consumer surplus from each customer. Holy Segments, Batman, how much money could we make if we ask every customer to tell us their maximum willingness to pay and then charge them that?

Surreal! Almost $80K! That's almost double the profits we made having just one price! Capturing the consumer surplus is clearly quite profitable. Even the 350 annoying people who only want to give me $49 each are making some contribution to the profits. All these customers are happy because we're asking them to pay the amount they were willing to pay already, so it's not like we're ripping anyone off. Kind of.

Here are some examples of segmentation you're probably familiar with:

  • Senior citizen discounts, since older people tend to be living on a "fixed income" and are willing to pay less than working-age adults
  • Cheap afternoon movie matinees (useful only to people without jobs)
  • Bizarre airfares, where everyone seems to be paying a different price. The secret about airfares is that people who are flying on business get their company to reimburse them, so they couldn't care less how much the ticket costs, while leisure travellers are spending their own money and they won't go if it costs too much.

Of course, the airlines can't just ask you if you're travelling on business, because pretty quickly everyone would catch on and lie to get the cheaper fares. But business travellers almost always travel on weekdays, and they hate spending a weekend away from home. So the airlines instituted policies that if you're staying over a Saturday night, you're probably not travelling on business, and they give you much cheaper fares if you stay over a Saturday night.

    There are more subtle ways to segment. You know those grocery coupons you see in the paper? The ones that get you 25 cents off a box of Tide detergent if you clip them out and remember to bring them to the store? Well, the trouble with grocery coupons is that there's so much manual labour involved in clipping them, and sorting them out, and remembering which ones to use, and choosing brands based on which coupons you have, and so on, and the net effect is that if you clip coupons you're probably working for about $7.00 an hour.

    Now, if you're retired and living off of social security, $7 an hour sounds pretty good, so you do it, but if you're a stock analyst at Merrill Lynch getting paid $12,000,000 a year to say nice things about piece-of-junk Internet companies, working for $7 an hour is a joke, and you're not going to clip coupons. Heck, in one hour you could issue "buy" recommendations on ten piece-of-junk Internet companies! So coupons are a way for consumer products companies to charge two different prices and effectively segment their market into two. Mail-in rebates are pretty much the same as coupons, with some other twists like the fact that they reveal your address, so you can be direct marketed to in the future.

    There are other ways to segment. You can market your products under different brand names (Old Navy vs. The Gap vs. Banana Republic) and hope that the rich people conveniently remember to buy their clothes at the Banana while the po' people go to Old Navy. In case there was any risk of people forgetting and going to the wrong store, the Banana Republic stores are conveniently situated in neighborhoods full of $2,000,000 condos while the Old Navy store is near the train station where you haul your poor tired ass back to New Jersey after a day of hard manual labor.

    In the world of software, you can just make a version of your product called "Professional" and another version called "Home" with some inconsequential differences, and hope that the corporate purchasers (again, the people who are not spending their own money) will be too embarassed at the thought of using "Windows XP Home Edition" at work and they'll buy the Pro edition. Home Edition at work? Somehow that feels like coming to work in your pyjamas! Ick!

    Quick trick: if you're going to try to pull the segmenting idea, you're probably going to be better off offering a discount to certain users rather than trying to charge some users a premium. Nobody likes feeling ripped off: people would rather buy a $199 product for $99 than a $59 product for $79. Theoretically, people should be rational. $79 is less than $99. Realistically, they hate feeling like someone is ripping them off. They'd much rather feel like they're getting a bargain than feel like they're getting gouged.

    ANYWAY.

    That was the easy part.

    The hard part is that everything I just told you is sort of wrong.

    Working my way backwards, this business about segmenting? It pisses the heck off of people. People want to feel they're paying a fair price. They don't want to think they're paying extra just because they're not clever enough to find the magic coupon code. The airline industry got really, really good at segmenting and ended up charging literally a different price to every single person on the plane. As a result most people felt they weren't getting the best deal, and they didn't like the airlines. When a new alternative arose in the form of low cost carriers (Southwest, jetBlue, etc.) customers had no loyalty whatsover to the legacy airlines that had been trying to pick their pockets for all those years.

    And God help you if an A-list blogger finds out that your premium printer is identical to the cheap printer, with the speed inhibitor turned off.

    So, while segmenting can be a useful tool to "capture consumer surplus," it can have significant negative implications for the long term image of your product. Many a small software vendor has seen their revenues go up and the amount of customer bickering about price go way down when they eliminated coupons, discounts, deals, multiple versions and tiers. Somehow, it seems like customers would rather pay $100 when everyone else is paying $100 than pay $79 if they know there's someone out there who got it for $78. Heck, GM made a whole car company, Saturn, based on the principle that the offered price is fair and you don't have to bargain.

    Even assuming you're willing to deal with a long-term erosion of customer goodwill caused by blatant price discrimination, segmentation is just not that easy to pull off. First of all, as soon as your customers find out you're doing it, they'll lie about who they are:

    • Frequent business travellers rearranged their tickets to include dual Saturday-night stays. For example a consultant living in Pittsburgh and working in Seattle Monday through Thursday would buy a two week trip from Pittsburgh to Seattle and then a weekend trip home in the middle. Both trips included Saturday night stays, it was the same flights they would have taken anyway, just much cheaper.
    • Got an academic discount? Everyone who is even vaguely related to someone vaguely associated with academia will start using it.
    • If your customers talk amongst themselves, they're going to find out about the price you're offering the other people, and you'll find yourself forced to match the lowest prices for everyone. Especially the big corporate purchasers who theoretically should have the "maximum willingness to pay" since they represent rich customers. Corporations have full time purchasing departments staffed with people whose entire job is whittling down prices. These people go to conferences where they learn how to get the best price. They practice saying "no. cheaper." all day long in front of mirrors. Your sales guy doesn't stand a snowflake's chance in hell.

    There are two forms of segmentation that slightly-too-clever software companies engage in that are not such great ideas:

    Bad Idea #1: Site Licenses.

    The opposite of segmentation, really. I have certain competitors that do this: they charge small customers per-user but then there's a "unlimited" license at a fixed price. This is nutty, because you're giving the biggest price break precisely to the largest customers, the ones who would be willing to pay you the most money. Do you really want IBM to buy your software for their 400,000 employees and pay you $2000? Hmm?

    As soon as you have an "unlimited" price, you are instantly giving a gigantic gift of consumer surplus to the least price-sensitive customers who should have been the cash cows of your business.

    Bad Idea #2: How Much Money Do You Have? Pricing.

    This is the kind used by software startups founded by ex-Oracle salesmen where the price isn't on the website anywhere. No matter how much you search to find the price, all you get is a form to provide your name, address, phone number, and fax number, for some reason, not that they're ever going to fax you anything.

    It's pretty obvious here that the plan is to have a salesman call you up and figure out how much you're worth, and then charge you that much. Perfect segmentation!

    This doesn't work so good either. First of all, the low end buyers are just going to move on. They will assume that if the price isn't listed, they can't afford it. Second, the people who don't like salesmen harassing them will just move on.

    Worse, as soon as you send a message that your price is negotiable, you're going to end up reverse segmenting. Here's why: the big companies you sell to, the ones who should be willing to give you the most money, are incredibly sophisticated about purchasing. They'll notice that your sales guy or gal is working on commission, and they'll know that he's got quarterly quotas, and they'll know that both the salesperson and the company are going to be incredibly desperate to make a sale at the end of the quarter (the salesperson to get his commission, and the company to avoid getting their knees shot off by their VCs or Wall Street). So the big customers will always wait until the last day in the quarter and end up getting a ridiculously good price which somehow involves weird accounting shenanigans so the company can book a lot of revenue that they're never really going to get.

    So, don't do site licenses, and don't try to make up prices as you go along.

    BUT WAIT!

    Do you really want to maximize profits? I glossed over something. You don't necessarily care about maximizing profits this month. You really care about maximizing all your profits, over time, in the future as well. Technically, you want to maximize the NPV of the stream of all future profits (without ever having your cash reserves dip below zero).

    Diversion: What's an NPV?

    What's worth more, $100 today or $100 in one year?

    Obviously $100 today, because you can invest it, say, in bonds, and at the end of the year you'll have, like, $102.25.

    So when you're comparing the value of $100 in one year to $100 today, you need to discount the $100 based on some interest rate. If the interest rate is 2.25%, for example, those $100 in the future should be discounted to $97.80, which is called the net present value (NPV) of $100 one year in the future.

    Go even further into the future, and you need to discount even more. $100 in five years, at today's interest rates, is worth only $84 today. $84 is the net present value of $100 in five years.

    Which would you rather earn?

    OPTION ONE:
    $5000, $6000, $7000 over the next three years

    OPTION TWO:
    $4000, $6000, $10000 over the next three years

    Option two sounds like a better deal, even after discounting the future earnings. If you take the second option, it's like investing $1000 in year one and getting $3000 back two years later, which is a very respectable investment!

    The reason I bring this up is because software is priced three ways: free, cheap, and dear.

    1. Free. Open source, etc. Not relevant to the current discussion. Nothing to see here. Move along.
    2. Cheap. $10 - $1000, sold to a very large number of people at a low price without a salesforce. Most shrinkwrapped consumer and small business software falls into this category.
    3. Dear. $75,000 - $1,000,000, sold to a handful of rich big companies using a team of slick salespeople that do six months of intense PowerPoint just to get one goddamn sale. The Oracle model.

    All three methods work fine.

    Notice the gap? There's no software priced between $1000 and $75,000. I'll tell you why. The minute you charge more than $1000 you need to get serious corporate signoffs. You need a line item in their budget. You need purchasing managers and CEO approval and competitive bids and paperwork. So you need to send a salesperson out to the customer to do PowerPoint, with his airfare, golf course memberships, and $19.95 porn movies at the Ritz Carlton. And with all this, the cost of making one successful sale is going to average about $50,000. If you're sending salespeople out to customers and charging less than $75,000, you're losing money.

    The joke of it is, big companies protect themselves so well against the risk of buying something expensive that they actually drive up the cost of the expensive stuff, from $1000 to $75000, which mostly goes towards the cost of jumping all the hurdles that they set up to insure that no purchase can possibly go wrong.

    Now, a quick glance around the Fog Creek website reveals that I'm firmly in camp #2. Why? Selling software at a low price means that I can get thousands of customers right away, some small, some large. And all those customers are going to be out there using my software and recommending it to their friends. When those customers grow, they'll buy more licenses. When people working at those customers move to new companies, they'll recommend my software to those new companies. Effectively I am willing to accept a lower price now in exchange for creating grassroots support. I see the low price of FogBugz as being an investment in advertising that I expect will pay off many times over in the long run. So far, it's working very well: FogBugz sales have grown more than 100% for three years without marketing, solely based on word-of-mouth and existing customers buying additional licenses.

    By comparison, look at BEA. Big company. Big price tag. The price alone means almost nobody has experience with their product. Nobody comes out of college and starts a dotcom using BEA technology, because they couldn't afford BEA technology in college. A lot of other good technologies have doomed themselves with high prices: Apple WebObjects was irrelevant as an application server because it started at $50,000. Who cared how good it was? Nobody ever used it! Anything made by Rational. The only way these products get into the hands of users is with an expensive full-frontal sales pitch. At these prices, the sales pitch is made to the executive, not the techie. The techies may well actively resist bad technology with good sales that the executives force down their throats. We have lots of FogBugz customers who have high-priced Remedy, Rational, or Mercury products sitting on the shelves after investments of well over $100,000, because that software isn't good enough to actually use. Then they buy a couple of thousand dollars worth of FogBugz and that's the product they really use. The Rational salesperson is laughing at me, because I have $2000 in the bank and he has $100,000. But I have far more customers than he does, and they're all using my product, and evangelizing it, and spreading it, while Rational customers either (a) don't use it or (b) use it and can't stand it. But he's still laughing at me from his 40 foot yacht while I play with rubber duckies in the bathtub. Like I said, all three methods work fine. But cheaper prices is like buying advertising and as such is an investment in the future.

    OK.

    Where was I.

    Oh yeah, before I started frothing at the mouth, I was picking apart the logic of deriving a demand curve. When I walked you through that whole discussion of the demand curve, you were probably asking yourself, "How do I know how much people are willing to pay?"

    You're right.

    That's a problem.

    You can't really find out what the demand curve is.

    You can have focus groups and ask people, but they'll lie to you. Some people will lie to show off their generosity and wealth. "Heck, yeah, I'd buy a pair of $400 jeans in a New York Minute!" Other people will lie because they really want your thing and they think you'll decide to charge less money if they tell you a low number. "Blogging software? Hmm. I'd pay, at most, 38 cents."

    Then you ask another focus group the next day, and this time, the first man to speak has a crush on a pretty woman in the group, and he wants to impress her, so he starts talking about how much his car cost and everyone is thinking Big Numbers. And the day after that, you serve Starbucks during the break, and while you're in the john everyone unbeknownst to you gets into a side conversation about paying $4 for a cup of coffee, and they're in a real frugal mood when you ask them about their willingness to pay.

    Then you finally get the focus group to agree that your software is worth $25 a month, and then you ask them how much they would pay for a permanent license and the same people just won't go a penny over $100. People seriously can't count.

    Or you ask some aircraft designers how much they would pay and they sort of think $99 would be a maximum price, even though aircraft designers regularly use software that costs on the order of $3000 a month without being aware of it, because someone else does the purchasing.

    So from day to day you get radically, and I mean radically different answers when you ask people how much they're willing to pay for something. The truth is, the only way to determine how much someone will pay for something is to put it up for sale, and see how many people actually buy it.

    Then you can try twiddling the prices to measure price sensitivity and try to derive the demand curve, but until you have something like 1,000,000 customers and you are absolutely sure that customer A will not find out you are offering a lower price to customer B, you will not get statistically meaningful responses.

    There's a real strong tendency to assume that experiments done on large populations of people should work out just like experiments done with chemicals in a high school lab, but everyone that has ever tried to do experiments on people knows that you get wildly variable results that just aren't repeatable and the only way you can be confident in your results is to carefully avoid ever doing the same experiment twice.

    And, in fact, you can't even be sure that the demand curve is downward sloping.

    The only reason we assumed that the demand curve is downward sloping is that we assumed things like "if Freddy is willing to buy a pair of sneakers for $130, he is certainly willing to buy those same sneakers for $20." Right? Ha! Not if Freddy is an American teenager! American teenagers would not be caught dead in $20 sneakers. It's, like, um, the death penalty? if you are wearing sneakers? that only cost $20 a pair? in school?

    I'm not joking around here: prices send signals. Movies in my town cost, I think, $11. Criminy. There used to be a movie theatre that had movies for $3. Did anyone go there? I DON'T THINK SO. It's obviously just a dumping ground for lousy movies. Somebody is now at the bottom of the East River with $20.00 cement sneakers because they dared to tell the consumer which movies the industry thought were lousy.

    You see, people tend to believe that you get what you pay for. The last time I needed a lot of hard drive space I invested in some nice cheap hard drives allegedly designed by Mr. Porsche himself that went for about $1 a gigabyte. Within six months all four had failed. Last week I replaced them with Seagate Cheetah SCSI hard drives that cost about $4 a gigabyte because I've been running those since I started Fog Creek four years ago without a glitch. Chalk it up to "you get what you pay for."

    There are just too many examples where you actually do get what you pay for, and the uninformed consumer is generally going to infer that the more expensive product is better. Buying a coffee maker? Want a really good coffee maker? You have two choices. Find the right issue of Consumer Reports in the library, or go to Williams-Sonoma and get the most expensive coffee maker they have there.

    When you're setting a price, you're sending a signal. If your competitor's software ranges in price from about $100 to about $500, and you decide, heck, my product is about in the middle of the road, so I'll sell it for $300, well, what message do you think you're sending to your customers? You're telling them that you think your software is "eh." I have a better idea: charge $1350. Now your customers will think, "oh, man, that stuff has to be the cat's whiskers since they're charging mad coin for it!"

    And then they won't buy it because the limit on the corporate AMEX is $500.

    Misery.

    The more you learn about pricing, the less you seem to know.

    I've been nattering on about this topic for well over 5000 words and I don't really feel like we're getting anywhere, you and I.

    Some days it seems like it would be easier to be a taxi driver, with prices set by law. Or to be selling sugar. Plain ol' sugar. Yep. That would be sweet.

    Take my advice, offered about 20 pages back: charge $0.05 for your software. Unless it does bug tracking, in which case the correct price is $30,000,000. Thank you for your time, and I apologize for leaving you even less able to price software than you were when you started reading this.


CityDesk Entity Classes

Friday, January 03, 2003

Comment: This article is Fog Creek Internal documentation, likely to be of interest only to hard core VB 6.0 geeks working with databases through DAO. Rather than use the DAO classes directly and scattering SQL syntax throughout our code, we try to isolate database access by using a thin layer which provides simple objects that map to underlying data. These objects are called "entities."

About 99% of what we do with databases consists of:

  • select one record
  • enumerate records, possibly matching a query
  • change a record
  • add a new record

The goal of entity classes is to make these things easy and consistent.

To isolate and simplify all access to the CityDesk database, all database access code goes through a set of classes called entity classes. These are VB user-defined classes, which are easily identified because the class names always start with the capital letter E followed by the name of the table. For example, you would use an entity class called EArticle to access the table tblArticle. There does not have to be an underlying physical table for each entity class. If you need to do a sophisticated query that joins multiple tables, this can be done inside an entity which still starts with E and has some sensible name but which does not correspond to a physical table.

Golden Rule: Never let entity classes cause you to do something inefficient with the database.

Comment: A lot of times people using the entity classes forget about what's involved in the underlying query, and end up writing code that's ridiculously inefficient. For example, updating a table by looping through it and committing each change, rather than using a SQL UPDATE statement. This is explicitly forbidden.

Each entity class has public members for each of the columns in the table with the exact same names. For example, for tblArticle, EArticle has ixArticle, ixSet, ixLanguage, etc. These are not implement as property get/lets, that just makes the code ugly; they are implemented as plain members. There is ONE EXCEPTION: any large or blob type fields are implemented as property get/lets which defer the actual database access until you access that particular field.

Comment: CityDesk has a lot of tables containing a large blob field, for example, the table of articles. If you're enumerating the articles in order to list them, you don't want to spend any cycles extracting the blobs.

When you create a new entity object, name it with the same name as the table in lower case, for example:

Dim article As EArticle : Set article = New EArticle

Creating a new entity object initializes all the fields to something sensible and initializes the primary key ix field (e.g. ixArticle) to -1.

Entity objects usually have these methods:

.Load(ix)

Loads the object from a single record, where the primary key is ix.

Dim article As EArticle : Set article = New EArticle
article.Load 1 ' load article where ixArticle = 1
Debug.Print article.ixLanguage

.Save

Commits any changes. If the primary key is still -1, this does an INSERT and always returns the just inserted primary key as a return value. If the primary key is not -1, this does an Update...Where and still returns the primary key.

' Add new article:
Dim article As EArticle : Set article = New EArticle
article.ixLanguage = 1
' set all the other fields
Dim ixArticleNew: ixArticleNew = article.Save

' Now edit it:
article.Load ixArticleNew
article.ixLanguage = 2
article.Save

Comment: We always use autonumber/identity fields for primary keys. One of the most maddening things about SQL is the numerous, inconsistent and unreliable ways of getting the identity field you just inserted.

.List
.GetNext

Used for enumerating through the records. The plain version of List starts enumerating them all. If there are zero matching records, that is, if EOF is true immediately, List returns False, in case you want to have different code for that case.

If you need to return any other kind of list, for example, all non-deleted, you use a variation of List like ListNonDeleted, which can take arguments.

Then .GetNext loads the first record and returns true if not EOF.

' List templates:
Dim template As ETemplate: Set template = New ETemplate
If template.List Then
While template.GetNext
Debug.Print template.Name
' No need to move next!
Wend
Else
Debug.Print "Nothing!"
End If

Comment: A common mistake in writing DAO code is forgetting the MoveNext at the bottom of your loop. This eliminates the need for it.


Colo Expansion Version 2.0

Saturday, February 05, 2005

(Updated 2/6/05)

It's been about two years since

In the past I would have upgraded the system, installed it all, and written an article about it. And then people would have emailed me to suggest better ways to do things, but it would be too late.

So now for the first time ever, I'm going to publish a "live" article here. So far, I haven't done anything. If you have any better suggestions for how to do things, it's not too late for me to learn from your experience. As we go along, I might have a few questions for my readers who have experience with this stuff. Thanks to everyone who has emailed me so far!

Step one was figuring out what problems we're trying to solve. Here's the list:

  • Not enough horsepower to handle the large number of FogBugz online trials we'll be hosting. FogBugz 4.0 is a couple of weeks away from shipping and that should create a flood of people wanting to take advantage of the free online trial. (By the way, if you want to beat the traffic, apply to join the FogBugz 4,0 beta.)
  • Running an email server on Windows allowed us to save hardware, at the cost of buying a commercial email product, IPswitch IMail. We are very close to using up the maximum number of mailboxes we're entitled to create, and we'd really like to switch to a Linux mail server running qmail or Postfix. The votes seem to be in favor of Postfix.
  • Although we have a standby server, a small Dell 750 we stuck in there last summer, we don't really have a good story for handling failures. Right now we could recover from most kinds of failures in about 24 hours, and we have to take all the sites down for about 15-30 minutes when we need to apply OS patches or upgrade the server. But Fog Creek's daily revenue has grown to the point where I'm not happy with that. Our new goal is to have no single point of hardware failure be able to bring down any of our sites, to be able to upgrade OSes without bringing everything down, and to be able to recover from very major problems in 15 minutes. This goal is fairly absurd. There will always be single points of failure. The ozone layer, for example. A more accurate goal would be "No single hard drive, power supply, or fan failure takes us out of commission for even 1 second. Any other server failure can be recovered from in 15 minutes. Routine Windows Updates will no longer bring our site down. If something happens to the colo facility, its power supply, or the City of New York, well, we go off the air for a while."
  • And we want to have an architecture in place to be able to scale up our hardware relatively easily to meet new demand.

Step one in this project was finding a good sysadmin to help design and implement this project. One of the great things about Fog Creek in its fifth year is that when we need to do something, we have the money to do it. So instead of me trying to build mission critical server networks using off-the-shelf consumer grade plastic linksys crap, I called up Adam Prato, who I had worked with at Juno, who has years of experience building out these kinds of things. After about an hour briefing he wrote up a very professional plan, sketched out the basic architecture of the new network, and started recommending purchases.

In choosing servers, there's a lot of benefit to using totally standard, identical parts. It makes it easy to swap things around when you need to scale or when something fails, and things are cheaper by the dozen. In order to avoid going totally broke, we still want reasonably affordable servers, but my requirement of "no single point of failure" means we need servers with dual power supplies, hot-swap RAID, and everything engineered for longetivity. That tends to narrow things down to the Dell PowerEdge 2850, which is the updated version of the 2650 we have now. I looked into the equivalent HP box. It looks like it costs 2-3 times as much for the equivalent thing.

The current plan is to get six servers:

1 and 2: web servers running Windows 2003. Two mirrors, each containing every site we host. We're using Win2003 instead of Win2000 because mirroring is much easier - the IIS 6.0 metabase is just a big XML file so we should be able to mirror the two machines using something as simple as ROBOCOPY. In fact, I think we can stage all our web sites back at the Fog Creek office and let Robocopy distribute them out to the web servers.

We will have some rudimentary load balancing by using a different IP address for every site that we host, and we can program each server's NIC to respond to some subset of IP addresses. That way moving a website from one server to the other is a simple matter of changing what IP addresses each server uses. We can scale up by adding more of these servers as we go along, and in the future we can use round-robin DNS or a real load balancer to distribute the load if necessary. If one server dies, the other one is a mirror so it can take up the whole load. Many people wrote in to recommend Windows' built-in Network Load Balancing. I think this may be overkill for the first generation but it sounds like a good idea in the long run.

3: SQL Server. Because everybody is sharing the same SQL Server, this machine will be slightly different than its peers: it will have the fastest available CPU available and it will have more hard drives plugged in. I heard that it's a good idea to get the transaction logs onto a different physical drive than the data itself, for better performance, so I think this machine will have two RAID 1 mirror sets (4 hard drives). Most correspondents agreed that this is the best architecture.

4: Mail and everything else server. Linux.

5: Routing server. Probably NetBSD. This machine will have four ethernet ports and will route between the public internet, the direct connection to the Fog Creek office, the "DMZ" containing our public-facing servers, and a private network segment containing the SQL Server machine. Many people suggested OpenBSD for better firewall support ("pf"). I'm out of my league here, and will have to trust Adam to use the right technology. Also having different OSes for #4 and #5 doesn't make sense so we'll try to pick one OS for both.

Other people thought that it might be better to use a solid-state hardware firewall/router rather than build our own on a generic Intel server. The advantages of building our own is a lot more flexibility, and somehow I think that the Dual-everything Dell PowerEdge 2850 is going to be a lot more robust than one of those dinky firewall/routers you get at CompUSA. I've already had one of those die on me with no explanation...

6: Cold backup. Another machine, identical to the first five, sitting around ready to be used for whatever we need it for or to serve as an emergency replacement.

Each machine will contain:

  • A Xeon processor, 800 MHz Front Side Bus, with 1 MB Cache
  • SCSI RAID with room for 6 drives. All the systems will start with 2x146GB drives installed in RAID 1 (mirroring) formation, except for the SQL machine which will have 4 drives. Every hard drive in the rack will be a Seagate Cheetah, 146GB SCSI U320, at 10000 rpm. Since they're hot swappable we can move them around at will and we can keep a couple of spares in the cage. 10K is much cheaper than 15K. Some people suggested using 15K for the SQL Server but I've actually heard elsewhere that RPMs aren't such a big deal for SQL Server. And I really don't want to have two kinds of hard drive.
  • Dual Gigabit NICS. The whole network is going to be gigabit ethernet and we'll use unmanaged Dell gigabit switches which seem to be priced well below the (NetGear) competition. The managed gigabit switches are more than 3x the price of unmanaged gigabit switches.
  • 2 GB RAM
  • Dell's DRAC 4/I remote management card (more about that in a minute).

I called up Dell and got a quote for this whole mess. By the way, if you're ever buying more than one or two machines, it's worth actually calling Dell. Somehow you always get a lower price than you would have gotten on the website. I've also been told that if you call during the last week of Dell's fiscal quarter, which I just missed, you get great deals as they spaz out trying to make their numbers for Wall Street in a classic display of American public company mismanagement.

I still have a list of open questions that I need to work on this week. If you have any answers, email is appreciated!

Question 1 - Power Management

Dell's built-in management card, the DRAC 4/I, has the ability to turn on and off the machine its plugged into, to resolve "stuck" systems, which is nice. But in two years of operating our current server, we've learned that the DRAC card itself (an older model, the DRAC 3) is more likely to freeze up than Windows 2000 itself. And there's no cure to a frozen DRAC card other than a full power cycle.

To address this, out of desperation, we put an APC MasterSwitch in the rack. This is basically a power strip with a web server. You can connect to it using any web browser to turn on or off anything plugged into the back. The MasterSwitch is awesome. It does one thing, and does it well.

The trouble is, this poor MasterSwitch only has 8 outlets and we have 6 servers with two power supplies each. It looks like there's a company called Sentry that makes something called the Dual Feed Power Tower, which, if I understand correctly, is designed exactly for these servers with dual feeds. Has anyone ever used such a thing? I can't find a price on the website which always makes me worry. It looks like it's about $1350. For less money I can get a new APC MasterSwitch Plus plus the "expansion pack" giving me 16 power outlets on two separate boxes thus reducing points of failure.

And another question -- according to Dell's very cool Rack Advisor, my six PowerEdge's are going to need about 35 amps of power. Yelps. Is that believeable? From what I remember you're not even allowed to have more than 20 amps on each circuit. So I'm wondering if a single power tower switcheroo thing can handle the load. And speaking of power, sometime this week I need to track down Joe Cooper of Peer 1 networks and find out how much it would cost to upgrade from a 1/8th rack to a 1/2 rack. We'll be going from 3U today to 14U. I'm hoping that Peer 1 has some way of getting me enough amps to run six servers. Or maybe when Dell says that one of their servers consumes over 5 amps, they mean "fully loaded", and we won't really be close to fully loaded. People have told me that Dell is on drugs. Technically the power supplies in these things are 700 watts, but realistically you never draw that much, and so 20 amps should be fine.

Question 2 - Rack Management

As soon as we got the second server, our small backup PowerEdge 750, and put it on top of the old 2650, we had to add RAM to the 2650 (on the bottom), which was a terrible pain in the buttocks. Which made me realize we really need to rack mount all our servers properly. I need to get Peer 1 to tell me what kind of mounting equipment I need to order... Dell gives you a choice of "VersaRails" or "RapidRails." I'm sure we'll figure out how to install things, but it would be nice if Dell had a video showing how to rack mount their servers instead of badly written technical documentation with instructions like "Install two 10-32 x 0.5-inch flange-head Phillips screws in the back mounting flange?euro;™s top and third-from-top holes to secure the slide assembly to the back vertical rail." What? VersaRail is for round holes and requires screwing things in. Rapid Rails is for square holes and things just click into place. It's not that hard, I'm told :)

Question 3 - Remote Control

We need to decide between a remote KVM-over-IP system and just using Dell's built-in DRAC 4/I remote control.

The DRAC has two advantages I know of:

  1. It has other nice features like power cycling (when it doesn't freeze) and the ability to boot your server off of a floppy or CD-ROM which is mounted over the network.
  2. It's a lot cheaper; KVM over IP is pretty expensive.

But I'm worried about the DRAC's KVM capabilities. With version 3, you could see the computer screen while it was booting up, but after that the remote control capability disappeared for a while until the operating system came up, after which it seemed to use Windows Terminal Services or something like that to remote control the server. This is a little bit to OS-dependent for comfort. I have heard rumors that the new model DRAC 4/I fixes this problem and lets you remote control the server no matter what state the OS is in. Can anyone confirm this? If so we can probably live without an expensive KVM-over-IP solution. The bottom line is that we need to have the DRAC 4/I, because there's no other way to turn on the server remotely. Power cycling a server that was on will make it come back on, but if you screw up and tell windows to shutdown you can power cycle until next monday and it won't go on. Given that we have to buy the DRAC 4/Is anyway, I'm hoping that they are good enough and we can live without expensive KVM over IP. If I'm wrong we'll just add KVM later.

Question 4 - SQL Server on its own Machine

I've never tried running SQL Server on a separate physical box than the ASP application that's using it. I know it works since FogBugz customers do it all the time, and it seems like it should improve performance, but then again, I'm worried about the network bottleneck. We'll have fast gigabit ethernet and the two machines are only one hop away. Do I need to worry about this? I don't have to worry about this, it works fine, etc. Many people pointed out that Best Practice is to have a separate network segment/switch for the SQL traffic so it doesn't compete with web server traffic. I strongly suspect this best practice arose in the days of 10/100 ethernet, not gigabit ethernet, and that this would be overkill, although if needed it will be easy to retrofit in later since every server has 2 ethernet ports.

Question 5 - RAID And *nix

People tell me that even using Dell-standard RedHat on their own servers with RAID has the problem that there's no way to find out when a drive has failed, and that the situation is even worse with the BSDs. This could be a major problem. There's no point in having RAID if you don't find out when a drive failed. We need to make sure whatever OS we choose for the two Unix boxes supports the Dell Perc Raid Thing.

FAQ

Q: Why Windows for the web servers?

A: FogBugz is written for Windows and ported to Unix. Thus the Windows version is usually about 2 months ahead of the Unix version, and we always want to be running the latest daily builds of our own dogfood. Also, a lot of Fog Creek's existing applications such as the online store are written for ASP or ASP.NET.

Q: Why SQL Server instead of MySQL?

A: Similar historical reasons, the performance is better, and the Unicode support is far better.

Q: Don't you like Linux/Unix/MacOSX/NetBSD/BeOS?

A: Yes. I love all operating systems equally. If we were building all this software from scratch I might have taken a different technical approach, but we didn't, so get off your OS religion high horse for a minute. We're going to have a mixed environment for the foreseeable future and 90% of FogBugz customers choose to run it on Windows servers. But we will use Unix for network routing, email, DNS, and a few other small things.

Q: Why Windows 2003 instead of Windows 2000?

A: Because IIS 6.0, the web server in Windows 2003, stores all its configuration information in a XML text-file metabase, and supports "XCOPY deployment," it means we'll be able to mirror web servers more trivially simply by mirroring file directories. And Windows 2003 has lots of improvements to web server performance.

Q: Why Dell?

A: My experience has been that it's easier to get a Dell server built and configured exactly the way you want it, and it usually comes out a lot cheaper than the alternatives. At least 2x cheaper than HP! And I don't like buying through resellers who don't know what they're selling. Every time I ever made the mistake of buying something from PC Connection I sentenced myself to weekly annoying sales calls from a clueless sales person who always dramatically screwed up anything I tried to buy from him. When you call Dell you get a nice, cheerful, competent kid in Austin, Texas who laughs at your jokes, no matter how dumb, and actually has the ability to do sophisticated things like read email that you sent him and then actually reply to it!

But that said, Dell has really, really crappy software skills, and their management software sucks big rocks. Every simple application they make is a horrible mess of Shockwave, HTML, Java, and Visual Basic with 35 MB of runtimes just to do some basic system administration thing. Today when I downloaded their otherwise cool Rack Advisor, the download was an EXE -- a self-extracting EXE. Running the EXE extracted -- wait for it -- another EXE. This EXE was the single file installer. Really, really, really crappy software skills. Get some programmers, Dell! HP and IBM are reputed to have much better system management software, not that I've ever seen it.

Finally, Dell has a neat feature where you can log on to their website and see a list of every computer you've ever bought from them, and I love logging on and seeing that list get longer and longer.

Q: Why not a SAN?

A: There seem to be lots of good reasons to use a storage area network, although I don't think it's quite cost-effective for our situation. We're looking at a total bill in the neighborhood of $20K, and SAN solutions tend to start around that price.


Command and Conquer and the Herd of Coconuts

Thursday, March 23, 2000

Recently I've been thinking a lot about what made Microsoft a great company to work for, and why working at Juno was so frustrating. At first I attributed the difference to the east coast/west coast thing, but I think it's a bit more subtle than that.

One of the most important things that made Microsoft successful was Bill Gates' devotion to hiring the best people. If you hire all A people, he said, they'll also hire A people. But if you hire B people, they'll hire the C people and then it's all over. This was certainly true at Microsoft. There were huge branches of the Microsoft tree filled with great people; these businesses were perennially successful (Office, Windows, and the developer products). But there were also branches that were just not as successful: MSN failed again and again and again; Microsoft Money took forever to get going, and Microsoft Consulting Services is full of airheads. In each of these cases it's pretty clear that a B leader built up a business unit full of C players and it just didn't work.

So: hire smart people who get things done. (See The Guerrilla Guide to Interviewing) It's so important there's a web page all about it and it's one of PaxDigita's most important goals. And when it worked, it worked quite well at Microsoft.

When I first went to interview at Juno, I wasn't sure that this was the company for me. The software looked a little bit goofy. The whole company smelled like a bunch of Wall Street Jocks who heard that the Internet was the New New Thing and they wanted in. So I called them up and said, "well, I'm not sure if I want to work there, but I'd like to learn more about your company."

In typical arrogant manner, they proceeded to schedule me for a whole battery of interviews and made me prove that I was good enough to work there. At first I was a little bit miffed, because I hadn't really decided that I was interested in the job in the first place. But then I realized that any company that has such a rigorous hiring process as Juno does has got to be full of smart people. This alone impressed me enough to take the job. Sure enough, Juno (and D. E. Shaw, the corporate parent) were full of geniuses. The receptionists had PhDs. The place was full of people like Dr. Eric Caplan, a former associate professor from U. Chicago who was writing technical documents for the intranet for heaven's sake. Talk about overqualified.

For a couple of years, I was very happy at Juno. It seemed like a dream job. (I even got interviewed for a website called dreamjobs.com talking about how dreamy Juno was). One thing was a little bit strange -- after about a year, I was promoted to be in charge of exactly 2 people on my little team of 6. In fact the average manager at Juno has about 2 reports, a stunningly vertical organization by any standard.

The next thing that I started noticing was strange is that a lot of people were leaving. In the wired wired world of Silicon Alley that's not uncommon. What was weird is how consistent their reason for leaving was: none of them felt any possibility of moving up in the company.

So what we had here was a company that was already as vertical and hierarchical as imaginable, with about 50 managers for a total of 150 employees, and people were complaining because there was no hope of moving up.

What's going on here?

At Microsoft, for some reason, lots of people at the lowest rung of the hierarchy were completely satisfied with their jobs and were happy to go on doing them forever. Whereas at Juno, people in the same positions rapidly got frustrated and wanted to leave because they couldn't get promoted.

I think the crucial difference here was in the corporate culture, specifically, in the way management worked.

At Microsoft, management was extremely hands-off. In general, everybody was given some area to own, and they owned it. Managers made no attempt to tell people what to do, in fact, they saw their role as running around, moving the furniture out of the way, so their reports could get things done.

There were some great examples of this. Managers always refused to resolve conflicts. Typically what would happen is that a designer would get into an argument with a developer over what a feature should look like. They would argue back and forth, discussing the issue for an hour, and eventually, failing to reach agreement, they would stomp into some manager's office hoping for a resolution. Now you've got three people in the room: a designer, a developer, and a manager. Who's the person who knows least about the problem? Obviously, it's the manager -- who was just hauled in at the last minute for Conflict Resolution. At Microsoft, the manager would usually refuse to make the decision. After all, they have the least information about the problem. The manager would generally force the designer and developer to work it out on their own, which, eventually, they did.

At Juno, quite the opposite was the case. Nobody at Juno owned anything, they just worked on it, and different layers of management happily stuck their finger into every pie, giving orders left and right in a style which I started calling hit and run management because managers tended to pop up unannounced, give some silly order for exactly how they wanted something done, dammit, without giving any thought to the matter, and leave the room for everyone else to pick up the pieces. The most egregious example was the CEO and president of the company, who would regularly demand printouts of every screen, take them home, and edit them using a red pen. His edits were sometimes helpful (spelling and grammar corrections), but usually, they demonstrated a complete lack of understanding as to what went into the screens and why they said what they said. For months later, we would have meetings where people would say things like "Charles [the CEO] doesn't like dropdown list boxes," because of something he had edited without any thought, and that was supposed to end the discussion. You couldn't argue against this fictional Charles because he wasn't there; he didn't participate in the design except for hit and run purposes. Ouch.

Hit and run management is but one symptom of what I would call Command and Control Management... something right out of the General Motors 1953 operations manual. In a particularly political company, it even becomes worse -- more like Command and Conquer management. It's completely inappropriate because it makes people unhappy, it causes the person with the least information to make the decisions, and it doesn't allow a corporation to take advantage of all the talents of the people it hired. If, like Juno, the corporation had been careful only to hire the brightest, most talented people, then it squandered an incredible resource and made those talented people frustrated as all hell.

When is Command and Conquer management acceptable? Well, it might work where your company is a Herd of Coconuts -- a bunch of underqualified morons who need herding. Perhaps something like the workfare corps that does the raking in Central Park. But software companies aren't Herds of Coconuts, and Command and Conquer doesn't cut it.

PaxDigita Culture

So this is why I'm concerned with creating the right culture of hands-off management at PaxDigita. In general:

  • everybody owns some area. When they own it, they own it. If a manager, or anybody else, wants to provide input into how that area is managed, they have to convince the owner. The owner has final say.
  • every decision is made by the person with the most information.
  • management is extremely flat. Ideally, managers just don't have time to get their fingers in the pies of their reports. You may be interested to read about a GE plant in North Carolina that has 170 employees who all report directly to the plant manager.

As a result, we will build a democratic culture where everybody runs the company. On a day to day basis, PaxDigita people have no boss -- they run themselves.


Contents of Joel on Software, the Book

Thursday, August 19, 2004

Contents of the Joel on Software book:

  • Introduction (all new!)
  • Choosing a Language
  • Back to Basics
  • The Joel Test: 12 Steps to Better Code
  • The Absolute Minimum Every Software Developer Absolutely, Positively Must Know About Unicode and Character Sets (No Excuses!)
  • Painless Functional Specifications
  • Painless Software Schedules
  • Daily Builds Are Your Friend
  • Hard-assed Bug Fixin'
  • Five Worlds
  • Paper Prototyping
  • Don't Let Architecture Astronauts Scare You
  • Fire And Motion
  • Craftsmanship
  • Three Wrong Ideas From Computer Science
  • Biculturalism
  • Every Crash, Everywhere
  • Interviewing
  • Incentive Pay Considered Harmful
  • Top Five (Wrong) Reasons You Don't Have Testers
  • Human Task Switches Considered Harmful
  • Things You Should Never Do, Part I
  • The Iceberg Secret, Revealed
  • The Law of Leaky Abstractions
  • Lord Palmerston on Programming
  • Measurement
  • Foreword to "In Search of Stupidity"
  • What is the Work of Dogs in this Country?
  • Getting Things Done When You're Only a Grunt
  • Two Stories
  • Big Macs vs. The Naked Chef
  • Nothing is as Simple as it Seems
  • In Defense of Not-Invented-Here Syndrome
  • Strategy Letter I: Ben and Jerry's vs. Amazon
  • Strategy Letter II: Chicken and Egg Problems
  • Strategy Letter III: Let Me Go Back!
  • Strategy Letter IV: Bloatware and the 80/20 Myth
  • Strategy Letter V: The Economics of Open Source
  • Murphy's Law Gone Wild
  • How Microsoft Lost the API War
  • Microsoft Goes Bonkers
  • Our .NET Strategy
  • Please Sir May I Have a Linker?
  • The Best of Ask Joel


Converting Capital Into Software That Works

Tuesday, March 21, 2000

I'm convinced that most people think about software companies in an upside-down way. The common belief is that when you're building a software company, the goal is to find a neat idea that solves some problem which hasn't been solved before, implement it, and make a fortune. We'll call this the build-a-better-mousetrap belief. But the real goal for software companies should be converting capital into software that works. If you understand this, it's easier to make the right strategic decisions.

The trouble with build-a-better-mousetrap is that there's not a lot of evidence that it works. First of all, many of the most successful software companies (Microsoft and Oracle, for example) don't really "innovate" in the sense that they are not really solving problems that haven't been solved before. In any market, it is exceedingly rare that you get to keep your invention to yourself. Everybody has competition. Wall Street Weenies and lawyers starting high tech companies tend to think they can protect themselves from this with patent protection. Ha! I can hardly think of a single case of a company successfully protecting themselves from competitors because of a patent. (Stac is the only case I can think of, and where the heck are they?)

The next problem with build-a-better-mousetrap is that we've reached a state with Internet software where there is too much money around chasing the same lame ideas. Call it the idrive-xdrive-swapdrive-freedrive phenomenon: suddenly thirty-seven companies pop up offering exactly the same service for free. There are a zillion examples of this. Petshops-on-the-net. Urban-video-rental-delivery. Cosmetics websites. When this happens, the business challenge switches from being a technical challenge that needs good programmers to being a marketing challenge that requires, somehow, the ability to break through the pack and establish a brand name, something that is vanishingly improbable. Not only that, but VC money is impatient. That means that investments which take a really long time to develop won't get funded, which is why anything really interesting or hard to copy won't even get funded. One reason that there are over thirty companies whose entire goal is providing free hard drive space on the Internet is that writing the code for such a service is so easy.

There's a different way to think of software development. Imagine that the goal of your software company is

Microsoft has a great recruiting strategy. They hire inexperienced, smart people right out of college... people that haven't learned from dysfunctional corporate cultures elsewhere. These kids get to Seattle, not really knowing anybody, work their asses off, and absorb the Microsoft way of doing things like a sponge. By the time they are ready to get disgruntled, their stock options start to kick in and their effective salary for staying around goes up into the stratosphere, where it is unlikely that anybody else will be able to lure them away. A five year Microsoft veteran could be making $500K a year including stock options -- care to match that?

The bottom line is that Microsoft is a better engine for converting cash into programmer productivity than anyone else. This is far from a peripheral factor. Most startups seem to think that hiring good programmers is about as important as, say, getting the right PR firm. They "know" it's important, but not how important. They're not really going to do what it takes to get and keep the best programmers. Which is why they are likely to get creamed. As soon as Netscape stopped producing code (their 5.0 browser is now something like 2 years behind schedule), their market share went away. WordPerfect's programmers believed that the best way to do things was Assembler and refused to code in C. Wrong idea, guys, it takes about 10 times as many programmers to do that, and now WordPerfect is a footnote. It's amazing how many companies - Lotus, Netscape, WordPerfect - have bitten the dust because they lost the ability to convert capital into code.

If you want to be the number one restaurant in town, you have to worry about how to get the best chefs and the best ingredients. You are a factory that converts raw ingredients and chefs to a dining experience. If you are a movie studio, you have to worry about getting the best actors, directors, and writers. You are a factory that converts talent into entertainment. So if you are a software company, you have to get the best chefs and directors: you are company that converts talent into code. The right talent knows how to make the right code which will make you successful.

OK, so let's assume-for-the-purpose-of-discussion that you agree with me, here. What implications does this have? In coming articles, I'll talk about putting this philosophy into practice: how to interview and choose the best people, how to create the right work environment that attracts people once you've picked them, and how you should basically subordinate all other problems to focusing on keeping the factory humming.


Craftsmanship

Monday, December 01, 2003

Making software is not a manufacturing process. In the 1980s everyone was running around terrified that Japanese software companies were setting up "software factories" that could churn out high quality code on an assembly line. It didn't make any sense then and it doesn't make sense now. Shoving a lot of programmers into a room and lining them up in neat rows did not really help get the bug counts down.

If writing code is not assembly-line style production, what is it? Some have proposed the label craftsmanship. That's not quite right, either, because I don't care what you say: that dialog box in Windows that asks you how you want your help file indexed does not in any way, shape, or form resemble what any normal English speaker would refer to as "craftsmanship."

Writing code is not production, it's not always craftsmanship (though it can be), it's design. Design is that nebulous area where you can add value faster than you add cost. The New York Times magazine has been raving about the iPod and how Apple is one of the few companies that knows how to use good design to add value. But I've talked enough about design, I want to talk about craftsmanship for a minute: what it is and how you recognize it.

I'd like to tell you about at a piece of code I've been rewriting for CityDesk 3.0: the file import code. (Advertainment: CityDesk is my company's easy-to-use content management product.)

The spec seems about as simple as any snippet of code can be. The user chooses a file using a standard dialog box, and the program copies that file into the CityDesk database.

This turned out to be a great example of one of those places where "the last 1% of the code takes 90% of the time." The first draft of the code looked like this:

  1. Open the file
  2. Read it all into a big byte array
  3. Store the byte array in a record

Worked great. For reasonable sized existing files it was practically instantaneous. It had a few little bugs, which I worked through one at a time.

The big bug surfaced when I stress-tested it by dragging a 120 MB file into CityDesk. Now, it is not common by any means for people to post 120 MB files on their web sites. In fact, it's quite rare. But it's not impossible, either. The code worked but took almost a minute and provided no visual feedback -- the app just froze and appeared to be completely locked up. This is obviously not ideal.

From a UI perspective, what I really wanted was for long operations to bring up a progress bar of some sort, along with a Cancel button. In the ideal world you would be able to continue doing other operations with CityDesk while the file copy proceeded in the background. There were three obvious ways to do this:

  1. From a single thread, polling frequently for input events
  2. By launching a second thread and synchronizing it carefully
  3. By launching a second process and synchronizing it less carefully

My experience with #1 is that it never quite works. It is too hard to ensure that all the code throughout your application can be run safely while a file copy operation is in progress. And Eric S. Raymond has convinced me that threads are usually not as good a solution as separate processes: indeed years of experience have shown me that programming with multiple threads creates much additional complexity and introduces whole new categories of dangerously frightful heisenbugs. #3 seemed like a good solution, especially since our underlying database is multiuser and doesn't mind lots of processes banging on it at the same time. So that's what I'm planning to do when I get back from Thanksgiving vacation.

Notice, though, the big picture. We've gone from read the file/save it in the database to something significantly more complicated: launch a child process, tell it to read the file and save it in the database, add a progress bar and cancel button to the child process, and then some kind of mechanism so the child can notify the parent when the file has arrived so it can be displayed. There will also be some work passing command line arguments to the child process, and making sure the window focus behaves in an expected manner, and handling the case of the user shutting down their system while a file copy is in progress. I would guesstimate that when all is said and done I'll have ten times as much code to handle large files gracefully, code that maybe 1% of our users will ever see.

And of course, a certain type of programmer will argue that my new child-process architecture is inferior to the original. It's "bloated" (because of all the extra lines of code). It has more potential for bugs, because of all the extra lines of code. It's overkill. It's somehow emblematic of why Windows is an inferior operating system, they will say. What's all this about progress indicators? they sneer. Just hit Ctrl+Z and then "ls -l" repeatedly and watch to see if the file size is growing!

The moral of the story is sometimes fixing a 1% defect takes 500% effort. This is not unique to software, no sirree, now that I'm managing all these construction projects I can tell you that. Last week, finally, our contractor finally put the finishing touches on the new Fog Creek offices. This consisted of installing shiny blue acrylic on the front doors, surrounded by aluminium trim with a screw every 20 cm. If you look closely at the picture, the aluminium trim goes all the way around each door. Where the doors meet, there are two pieces of vertical trim right next to each other. You can't tell this from the picture, but the screws in the middle strips are almost but not exactly lined up. They are, maybe, 2 millimeters off. The carpenter working on this measured carefully, but he was installing the trim while the doors were on the ground, not mounted in place, and when the doors were mounted, "oops," it became clear that the screws were not exactly lined up.

This is probably not that uncommon; there are lots of screws in our office that don't line up perfectly. The problem is that fixing this once the holes are drilled would be ridiculously expensive. Since the correct placement for the screws is only a couple of millimeters away, you can't just drill new holes in the door; you'd probably have to replace the whole door. It's just not worth it. Another case where fixing a 1% defect takes 500% effort, and it explains why so many artifacts in our world are 99% good, not 100% good. (Our architect never stops raving about some really, really expensive house in Arizona where every screw lined up.)

It comes down to an attribute of software that most people think of as craftsmanship. When software is built by a true craftsman, all the screws line up. When you do something rare, the application behaves intelligently. More effort went into getting rare cases exactly right than getting the main code working. Even if it took an extra 500% effort to handle 1% of the cases.

Craftsmanship is, of course, incredibly expensive. The only way you can afford it is when you are developing software for a mass audience. Sorry, but internal HR applications developed at insurance companies are never going to reach this level of craftsmanship because there simply aren't enough users to spread the extra cost out. For a shrinkwrapped software company, though, this level of craftsmanship is precisely what delights users and provides longstanding competitive advantage, so I'll take the time and do it right. Bear with me.


Daily Builds Are Your Friend

Saturday, January 27, 2001

In 1982, my family took delivery of the very first IBM-PC in Israel. We actually went down to the warehouse and waited while our PC was delivered from the port. Somehow, I convinced my dad to get the fully-decked out version, with two floppy disks, 128 K memory, and both a dot-matrix printer (for fast drafts) and a Brother Letter-Quality Daisy Wheel printer, which sounds exactly like a machine gun when it is operating, only louder. I think we got almost every accessory available: PC-DOS 1.0, the $75 technical reference manual with a complete source code listing of the BIOS, Macro Assembler, and the stunning IBM Monochrome display with a full 80 columns and ... lower case letters! The whole thing cost about $10,000 including Israel's then-ridiculous import taxes. Extravagant!

Now, "everybody" knew that BASIC was a children's language that requires you to write spaghetti code and turns your brain into Camembert cheese. So we shelled out $600 for IBM Pascal, which came on three floppy diskettes. The compiler's first pass was on the first diskette, the second pass was on the second diskette, and the linker was on the third diskette. I wrote a simple "hello, world" program and compiled it. Total time elapsed: 8 minutes.

Hmm. That's a long time. I wrote a batch file to automate the process and shaved it down to 7 1/2 minutes. Better. But when I tried to write long programs like my stunning version of Othello which always beat me, I spent most of the time waiting for compiles. "Yep," a professional programmer told me, "we used to keep a sit-up board in the office and do sit-ups while we were doing compiles. After a few months of programming I had killer abs."

One day, a spiffy program called Compas Pascal appeared from Denmark, which Philippe Kahn bought and renamed Borland Turbo Pascal. Turbo Pascal was sort of shocking, since it basically did everything that IBM Pascal did, only it ran in about 33K of memory including the text editor. This was nothing short of astonishing. Even more astonishing was the fact that you could compile a small program in less than one second. It's as if a company you had never heard of introduced a clone of the Buick LeSabre which could go 1,000,000 MPH and drive around the world on so little gasoline than an ant could drink it without getting sick.

Suddenly, I became much more productive.

That's when I learned about the concept of the REP loop. REP stands for "Read, Eval, Print", and it describes what a lisp interpreter does for a living: it "reads" your input, evaluates it, and prints the result. An example of the REP loop is shown below: I type something, and the lisp interpreter reads it, evaluates it, and prints the result.

On a slightly larger scale, when you're writing code, you are in a macro-version of the REP loop called the Edit-Compile-Test loop. You edit your code, compile it, test it, and see how well it works.

A crucial observation here is that you have to run through the loop again and again to write a program, and so it follows that the faster the Edit-Compile-Test loop, the more productive you will be, down to a natural limit of instantaneous compiles. That's the formal, computer-science-y reason that computer programmers want really fast hardware and compiler developers will do anything they can to get super-fast Edit-Compile-Test loops. Visual Basic does it by parsing and lex-ing each line as you type it, so that the final compile can be super-quick. Visual C++ does it by providing incremental compiles, precompiled headers, and incremental linking.

But as soon as you start working on a larger team with multiple developers and testers, you encounter the same loop again, writ larger (it's fractal, dude!). A tester finds a bug in the code, and reports the bug. The programmer fixes the bug. How long does it take before the tester gets the fixed version of the code? In some development organizations, this Report-Fix-Retest loop can take a couple of weeks, which means the whole organization is running unproductively. To keep the whole development process running smoothly, you need to focus on getting the Report-Fix-Retest loop tightened.

One good way to do this is with daily builds. A daily build is an automatic, daily, complete build of the entire source tree.

Automatic - because you set up the code to be compiled at a fixed time every day, using cron jobs (on UNIX) or the scheduler service (on Windows).

Daily - or even more often. It's tempting to do continuous builds, but you probably can't, because of source control issues which I'll talk about in a minute.

Complete - chances are, your code has multiple versions. Multiple language versions, multiple operating systems, or a high-end/low-end version. The daily build needs to build all of them. And it needs to build every file from scratch, not relying on the compiler's possibly imperfect incremental rebuild capabilities.

Here are some of the many benefits of daily builds:

  1. When a bug is fixed, testers get the new version quickly and can retest to see if the bug was really fixed.
  2. Developers can feel more secure that a change they made isn't going to break any of the 1024 versions of the system that get produced, without actually having an OS/2 box on their desk to test on.
  3. Developers who check in their changes right before the scheduled daily build know that they aren't going to hose everybody else by checking in something which "breaks the build" -- that is, something that causes nobody to be able to compile. This is the equivalent of the Blue Screen of Death for an entire programming team, and happens a lot when a programmer forgets to add a new file they created to the repository. The build runs fine on their machines, but when anyone else checks out, they get linker errors and are stopped cold from doing any work.
  4. Outside groups like marketing, beta customer sites, and so forth who need to use the immature product can pick a build that is known to be fairly stable and keep using it for a while.
  5. By maintaining an archive of all daily builds, when you discover a really strange, new bug and you have no idea what's causing it, you can use binary search on the historical archive to pinpoint when the bug first appeared in the code. Combined with good source control, you can probably track down which check-in caused the problem.
  6. When a tester reports a problem that the programmer thinks is fixed, the tester can say which build they saw the problem in. Then the programmer looks at when he checked in the fix and figure out whether it's really fixed.

Here's how to do them. You need a daily build server, which will probably be the fastest computer you can get your hands on. Write a script which checks out a complete copy of the current source code from the repository (you are using source control, aren't you?) and then builds, from scratch, every version of the code that you ship. If you have an installer or setup program, build that too. Everything you ship to customers should be produced by the daily build process. Put each build in its own directory, coded by date. Run your script at a fixed time every day.

  • It's crucial that everything it takes to make a final build is done by the daily build script, from checking out the code up to and including putting the bits up on a web server in the right place for the public to download (although during the development process, this will be a test server, of course). That's the only way to insure that there is nothing about the build process that is only "documented" in one person's head. You never get into a situation where you can't release a product because only Shaniqua knows how to create the installer, and she was hit by a bus. On the Juno team, the only thing you needed to know to create a full build from scratch was where the build server was, and how to double-click on its "Daily Build" icon.
  • There is nothing worse for your sanity then when you are trying to ship the code, and there's one tiny bug, so you fix that one tiny bug right on the daily build server and ship it. As a golden rule, you should only ship code that has been produced by a full, clean daily build that started from a complete checkout.
  • Set your compilers to maximum warning level (-W4 in Microsoft's world; -Wall in gcc land) and set them to stop if they encounter even the smallest warning.
  • If a daily build is broken, you run the risk of stopping the whole team. Stop everything and keep rebuilding until it's fixed. Some days, you may have multiple daily builds.
  • Your daily build script should report failures, via email, to the whole development team. It's not too hard to grep the logs for "error" or "warning" and include that in the email, too. The script can also append status reports to an HTML page visible to everyone so programmers and testers can quickly determine which builds were successful.
  • One rule we followed on the Microsoft Excel team, to great effect, was that whoever broke the build became responsible for babysitting builds until somebody else broke it. In addition to serving as a clever incentive to keep the build working, it rotated almost everybody through the job of buildmaster so everybody learned about how builds are produced.
  • If your team works in one time zone, a good time to do builds is at lunchtime. That way everybody checks in their latest code right before lunch, the build runs while they're eating, and when they get back, if the build is broken, everybody is around to fix it. As soon as the build is working everybody can check out the latest version without fear that they will be hosed due to a broken build.
  • If your team is working in two time zones, schedule the daily build so that the people in one time zone don't hose the people in the other time zone. On the Juno team, the New York people would check things in at 7 PM New York time and go home. If they broke the build, the Hyderabad, India team would get into work (at about 8 PM New York Time) and be completely stuck for a whole day. We started doing two daily builds, about an hour before each team went home, and completely solved that problem.

For Further Reading:

  • Some discussion on tools for daily builds
  • Making daily builds is important enough that it's one of the 12 steps to better code.
  • There's a lot of interesting stuff about the builds made (weekly) by the Windows NT team in G. Pascal Zachary's book Showstopper.
  • Steve McConnell writes about daily builds here.


Documentary Filmmaker Wanted

Wednesday, March 23, 2005

We are looking for a talented filmmaker, student or experienced, to make a documentary about the software development process this summer. If you think you're interested, read on for more details!

This summer, Fog Creek Software has hired four summer interns from Yale, Duke, and Rose-Hulman. Our selection process was extremely competitive, with over 800 kids applying for only four positions.

Instead of wasting their talents giving them the usual dull and unimportant tasks of a typical summer internship, we decided to let the interns create a complete new software product, from beginning to end, over the course of one summer. With experienced software developers as mentors, the team will design, program, test, and roll out a complete software product over the course of one hectic summer, going from concept to paying customers in about ten weeks.

This project would be an excellent opportunity for a documentary filmmaker to create a film about the software development process, in the spirit of Code Rush, Triumph of the Nerds, Startup.com, etc. We've been thinking of it as a kind of serious Apprentice-meets-Real-World. Unlike The Apprentice, our business challenge will be a serious ten week project that becomes a real product, not four hours of wrapping chocolate bars disguised as a "business challenge." And we don't have a crazy boss with a bad combover firing people every week.

Why would I want to make this film?

We think it will be extremely interesting to everyone who is curious about the process of developing software. It is a unique opportunity to film the complete software development process over the course of one summer in New York City. And it might serve as an excellent demo for your portfolio if you are interested in making documentaries, industrials, or working on a reality TV show.

How will the film be financed?

This is not an industrial; it's a documentary. We're looking for a filmmaker who will finance and make the film themselves and own the rights. We want someone who can promote the film to typical documentary outlets (TV, IFilm, cable, video, etc, or just as a student project for a class).

Producing the film on digital video is probably good enough, but again, it's your film, so it's up to you.

Fog Creek Software will make available a cash grant of $5000 plus up to $5000 in expense reimbursement to the filmmaker we chose to work with, conditional on producing a finished film. If you're a student and are willing to scrounge a little bit, this may be enough to actually make the film in DV. If you're a professional documentary filmmaker, you probably know how to raise money to make films better than we do.

Again, this is not intended to finance the film, and we do not expect to own the rights to the film; it's just a subsidy that may make it easier.

Who will see the film?

The founder of Fog Creek Software, Joel Spolsky, has his own website called Joel on Software which is extremely popular among software developers around the world. This site gets over 7,000,000 hits a month, has 350,000 unique visitors every month, and usually ranks first among personal web sites read by software developers. The Joel on Software audience is extremely interested in the software development process and will likely be very interested in this film. The interns will be maintaining a weblog all summer describing the development process, and building up a natural audience for a documentary about their experience.

When the documentary is ready, Fog Creek Software will make the DVD available on its website, serving as a traditional retailer and promoting the film extensively to Fog Creek customers and the Joel on Software audience. This will get your film a fairly decent audience, as documentaries of this type go, but we can't make any promises. You will probably want to cut a trailer as well which we will use to promote the film on our websites.

What's in the documentary?

You're the filmmaker. But here are some of the things you might want to film:

  • The project kick-off meeting and progress meetings
  • Interviews with the developers and the interns
  • Candid interviews with interns about their experience, before, during, and after the project
  • Installing new servers at our data center
  • We have a very photogenic, architect-designed loft office in the New York City fashion district
  • Other summer activities that our interns do (moving into summer housing, night on the town, New York touristy activities like visits to museums and the Circle Line which we will organize for the interns)
  • Fog Creek founder Joel Spolsky will be giving a speech to a large group of developers at a conference in Washington DC

The goal is to tell the story of a the summer internship and the new product development simultaneously, in the form of a fairly level-headed documentary. We're not making a reality TV program, and the participants' personal lives are their own.

What's the Schedule?

Filming will begin no later than May 15th. The interns will be moving in around May 29th. The development project will be finished by August 15th.

I'm Interested! Sign Me Up!

The only way to apply if you're interested is by sending a hard copy resume/CV, cover letter, and DVD portfolio of your work to Fog Creek Software as soon as possible:

Fog Creek Software
Project Aardvark
535 8th Ave., 18th Floor North
New York, NY 10018

Nothing you send us can be returned, and we cannot accept applications sent via email or any other method other than plain old mail. The application deadline is April 15th, 2005, which is really soon, so get on it!

Any Questions?

Email questions to Joel Spolsky, spolsky [at] fogcreek [dot] com.


Does Issuing Passports Make Microsoft a Country?

Wednesday, July 26, 2000

Am I the only one who is terrified about Microsoft Passport? It seems to me like a fairly blatant attempt to build the world's largest, richest consumer database, and then make fabulous profits mining it. It's a terrifying threat to everyone's personal privacy and it will make today's "cookies" seem positively tame by comparison. The scariest thing is that Microsoft is advertising Passport as if it were a benefit to consumers, and people seem to be falling for it! By the time you've read this article, I can guarantee that I'll scare you into turning off your Hotmail account and staying away from MSN web sites.

This article has two parts. First, I'll present a brief technical overview of how Passport works and why it eliminates the last line of defense protecting your privacy. Second, I'll talk about how Microsoft plans to develop Passport to create a massive consumer information database and link all your private information together, and how they plan to profit fantastically from it.

But before I get started, let me say that I'm not just writing this to bash Microsoft. That's not my goal here. Microsoft is a large, diverse company with many smart people and many ethical people; they have many great products and some pathetic products, too. I spent 3 years working at Microsoft, many of my friends are still there, and I'm a Microsoft shareholder. I'm writing this article because I think the Microsoft Passport story is fascinating from a privacy perspective and from a business strategy perspective, and because nobody else seems to be covering it.

1. How Passport Works

In the olden days of interactive computing, you got an account on one computer which was all you ever used. You had one username and one password to remember. The web has changed things dramatically: because it is so easy to visit lots of web sites, you may have accounts with dozens of companies on dozens of different computers. I have 81 at last count. Most people have no hope of remembering 81 different account names and passwords, so they tend to just use the same password on every site, or they keep a long list of passwords written down somewhere. It's a bit of a nuisance. If you regularly shop online, you're probably getting sick of typing in your home address, credit card information, and remembering the user name and password for all those sites. It's extremely common for people to abandon their shopping carts on the web when they see the long form they have to fill out to make an account and purchase their products.

This is the kind of problem that Passport is promising to solve. To understand how it works, I'd like to take a few minutes to talk about some web security and privacy technology and how Passport subverts it.

How Cookies Work

There's a lot of wrong information about cookies out there. All a cookie does is tell a web site operator when somebody comes back to their site that has been there before. It doesn't give the web site operator any information about that person's identity; it just says "Hey, that visitor who was here last Tuesday at 4:15 PM? That person is back again."

Technically, the way it works is that when you go to the web site for the first time, the web server makes up an ID for you, for example, if I go to www.eCrap.com, it might make up the number 76JU589SU for me, which is completely meaningless. The web server sends this meaningless ID number to my web browser, which stores it.

Now, the next time I go to eCrap.com, my web browser will tell the web server: "Yo, in case you care, this is 76JU589SU coming back again. Thought you might want to know."

That's all there is to it. Now, since eCrap is smart, they opened a file on me, marked 76JU589SU. In that file, they could keep any information I give them. If I buy something from eCrap and give them my address, they could store my address in their 76JU589SU file. And my credit card. And a list of the things I bought. Next time I wanted to buy something, since they already knew who I am, they can offer to let me purchase it without typing in an address or credit card number, because they can just look that up in their file.

Theoretically, the only information eCrap can put in their file is the information that I give them. Amazon's files probably contain information about what books I bought, my address, credit card information, and maybe some information about what books I looked at but didn't buy... any information that they can gather from my activity on their web site. Amazon does not know how old I am or what color my hair is, since I never told them that information. They don't know that my favorite cafe is The Big Cup in New York City, because I never gave them that information, either. But they do know that I bought the book 101 Cute Puppies from them. One day, if 101 More Cute Puppies comes out, they are probably going to search their files for people who bought 101 Cute Puppies and tell us about the sequel the next time we log on.

How Cookies Protect Your Privacy

Now, suppose I decide to open a credit card account online. Of course, the credit card company would probably love to know that I just bought "Bankruptcy for Dummies" and "How to Stiff Everyone And Move To Brazil" from Amazon.com, but they are not going to find out. Why? Because my web browser will simply never send my Amazon cookie to the credit card company. The golden rule of cookies is that they are only sent back to the same web domain as they came from. This is important to remember, because it's the only thing that really protects you from having all the web sites you visit swap information about you. I don't want my credit card company to know that I bought a bankruptcy book. I don't want potential landlords to know that I read lots of articles about caring for Boa Constrictors. I don't want potential employers to know that I read web sites about homemade bombs. They'll probably take it the wrong way.

Unfortunately, this is one case where the consumer's interests and the web site's interests are diametrically opposed. Every web site in the world wants to show you targeted ads. When I visit The Dilbert Zone, they would love to know that I read The Jerusalem Post online and send me an ad for luxury apartments in Israel, because targeted ads sell for a lot more money than non-targeted ads.

Subverting The Golden Rule

Web advertising companies, like Doubleclick, are trying to collect as much information about people as possible, so that they can send them targeted ads. The way they do this is by having their member sites show ads which come from the same web domain.

Here's an example of how this works: I go to The Jerusalem Post to read the latest news. The Jerusalem Post web site includes an advertisement which is actually served up by the Doubleclick web server. Now Doubleclick opens a file on me and sends a cookie back to my web browser.

Later that day, I go to the Dilbert Zone. Dilbert also includes an advertisement, also served up by the Doubleclick web server. Remember, The golden rule of cookies is that they are only sent back to the same web domain as they came from. So my naive web browser says, "Oh, you're going back to Doubleclick, I'll just tell them that you're the same person that was here before..." and now Doubleclick knows that somebody who went to The Jerusalem Post before is now visiting Dilbert, so they show me that ad for the expensive apartment in Israel.

Passport Has Another Way

The Doubleclick trick for sharing your information only works for ads, but Microsoft Passport found a way to work around the golden rule for any site. Here's how it works.

Go to http://www.hotmail.com. Watch what your web browser does. You'll see that your browser first goes to Hotmail for a second, then jumps to www.passport.com for a split second, and then immediately goes right back to Hotmail. What's going on?

It turns out that there's a feature to allow a web page to tell your browser to go somewhere else instead. For example, if you try to go to eCrap.com, that site might tell your web browser "Oh, we've gone bankrupt. Please go to our lawyer's site instead, DeweyCheatumAndHowe.com." It's called a client redirect.

That's what Hotmail is doing. It only takes a couple of seconds, but while it's happening, Hotmail and Passport are communicating through your web browser about who you are.

Now, if you go to another Microsoft web site, say, www.investor.com, the same thing will happen: you'll get redirected to Passport and then back to Investor. Because Passport is "telling on you", even though your web browser is supposed to be protecting your security by following the golden rule of cookies, it's really Passport that is signing you in. Bottom line: Hotmail knows that you're the same person that just went to Investor. And that applies to any Microsoft web site: Slate, Expedia, Hotmail, Investor, MSN, etc.

The way Passport uses client redirect to subvert cookie security is basically just taking advantage of a security hole in web browsers. Cookies weren't meant to allow this. But you can bet that this is one security bug that Microsoft is not going to fix.

To Summarize:

The golden rule of cookies that protects your privacy is that they are only sent back to the same web domain as they came from. Microsoft Passport eliminates this protection allowing any Passport site to share information about you.

2. How Passport Will Be Developed

The supposed benefit of Passport to consumers is that it allows them to use one login and password to access all the Passport web sites.

But the benefit to the web sites is much greater, because now they can pool and share their information about you. Let's take a hypothetical example that's possible today. Microsoft's online travel agency Expedia is a Passport web site, and Microsoft Investor is too. One day, Expedia could start offering higher fares to customers who have more than a million dollars in their Investor stock portfolio. There's not really anything technically impossible about this, and it's probably legal, too.

Web businesses would love to have a way to combine their files on you. And the more businesses that have the opportunity to combine their files, the more valuable it is. There's a network effect going on here (a.k.a. Metcalfe's Law): the value of a network of web sites who swap data is the square of the number of sites in the network, because every site can exchange data with every other site.

The spooky thing about Passport is that there's one company that serves as the gatekeeper to joining the network: Microsoft. Which is why this has the potential of being a phenomenally valuable business.

There are many ways Microsoft can profit from Passport. They could charge a commission when web sites sell data about consumers. They could sell private information which they collect from their participant sites. Or they could just charge web sites to belong to the network. It's a great business that makes credit agencies look like they have nothing.

The scary thing is that if you use Internet Explorer, Microsoft controls your web browser. You can be sure that Microsoft would love to eliminate that nasty two second flash while your web browser is redirected through passport.com. I'll bet there's a feature under development for a future version of IE that will make Passport just be built into the web browser, or even built into the operating system itself. Don't believe me? Here's a quote from Microsoft's .NET white paper:

Building on Microsoft Passport and Windows authentication technology, [Windows.NET] provides levels of authentication ranging from passwords and wallets to smart cards and biometric devices. Enables developers to build services that provide personalization and privacy for their customers, who in turn can enjoy new levels of safe and secure access to their services, no matter where they are or on what device. Supported in the first major release of Windows.NET, code-named "Whistler."

Notice the way Microsoft acts as if they are providing "privacy" and a "new level of safe and secure access." Uh huh. The best way to lie is through repeated assertion until eventually nobody notices the lie.

Passport will be built into IE, it will be built into the operating system, and it will be made available as a programming interface so that developers can use it, and frankly, there goes your last defense against corporations building up gigantic super-databases with outrageous amounts of personal information about everyone.

Yeah sure, Microsoft promises to protect your privacy... Does anybody really believe this for a minute? Every day there's a new story about a security breach -- Hotmail itself, a Passport site, had a major security breach a couple of months ago that made it into the headlines. During the next wave of web based business failures, we're going to start seeing a lot more stories like the one about how toysmart.com, as soon as they went bankrupt, reneged on their promise to protect the privacy of their customers. Even the best laid plans to protect consumer's privacy don't work. There are always software bugs and security goof-ups. Unscrupulous employees on the inside abuse their ability to look at the database. Court orders and subpoenas force companies to divulge information they promised to keep secret.

If Microsoft was honest about protecting your privacy, they would let users keep their private information on their own computers, and they would ask you every time they were going to reveal some data.

But they're not being honest. They want all your data in a big database on their server, thank you very much, and they want you to click "I Agree" to the 27 pages of legalese which says things like "Microsoft reserves the right to amend this agreement at any time." If you really trust any Internet company to protect your privacy, I've got a bridge to sell ya.

Read some of my reader's responses.
JunkBusters on Microsoft and Privacy


Don't Let Architecture Astronauts Scare You

Saturday, April 21, 2001

When great thinkers think about problems, they start to see patterns. They look at the problem of people sending each other word-processor files, and then they look at the problem of people sending each other spreadsheets, and they realize that there's a general pattern: sending files. That's one level of abstraction already. Then they go up one more level: people send files, but web browsers also "send" requests for web pages. And when you think about it, calling a method on an object is like sending a message to an object! It's the same thing again! Those are all sending operations, so our clever thinker invents a new, higher, broader abstraction called messaging, but now it's getting really vague and nobody really knows what they're talking about any more. Blah.

When you go too far up, abstraction-wise, you run out of oxygen. Sometimes smart thinkers just don't know when to stop, and they create these absurd, all-encompassing, high-level pictures of the universe that are all good and fine, but don't actually mean anything at all.

These are the people I call Architecture Astronauts. It's very hard to get them to write code or design programs, because they won't stop thinking about Architecture. They're astronauts because they are above the oxygen level, I don't know how they're breathing. They tend to work for really big companies that can afford to have lots of unproductive people with really advanced degrees that don't contribute to the bottom line.

A recent example illustrates this. Your typical architecture astronaut will take a fact like "Napster is a peer-to-peer service for downloading music" and ignore everything but the architecture, thinking it's interesting because it's peer to peer, completely missing the point that it's interesting because you can type the name of a song and listen to it right away.

All they'll talk about is peer-to-peer this, that, and the other thing. Suddenly you have peer-to-peer conferences, peer-to-peer venture capital funds, and even peer-to-peer backlash with the imbecile business journalists dripping with glee as they copy each other's stories: "Peer To Peer: Dead!"

The Architecture Astronauts will say things like: "Can you imagine a program like

Another common thing Architecture Astronauts like to do is invent some new architecture and claim it solves something. Java, XML, Soap, XmlRpc, Hailstorm, .NET, Jini, oh lord I can't keep up. And that's just in the last 12 months!

I'm not saying there's anything wrong with these architectures... by no means. They are quite good architectures. What bugs me is the stupendous amount of millennial hype that surrounds them. Remember the Microsoft Dot Net white paper?

The next generation of the Windows desktop platform, Windows.NET supports productivity, creativity, management, entertainment and much more, and is designed to put users in control of their digital lives.

That was about 9 months ago. Last month, we got Microsoft Hailstorm. That white paper says:

People are not in control of the technology that surrounds them....HailStorm makes the technology in your life work together on your behalf and under your control.

Oh, good, so now the high tech halogen light in my apartment will stop blinking randomly.

Microsoft is not alone. Here's a quote from a Sun Jini whitepaper:

These three facts (you are the new sys admin, computers are nowhere, the one computer is everywhere) should combine to improve the world of using computers as computers -- by making the boundaries of computers disappear, by making the computer be everywhere, and by making the details of working with the computer as simple as putting a DVD into your home theater system.

And don't even remind me of the fertilizer George Gilder spread about Java:

A fundamental break in the history of technology...

That's one sure tip-off to the fact that you're being assaulted by an Architecture Astronaut: the incredible amount of bombast; the heroic, utopian grandiloquence; the boastfulness; the complete lack of reality. And people buy it! The business press goes wild!

Why the hell are people so impressed by boring architectures that often amount to nothing more than a new format on the wire for RPC, or a new virtual machine? These things might be good architectures, they will certainly benefit the developers that use them, but they are not, I repeat, not, a good substitute for the messiah riding his white ass into Jerusalem, or world peace. No, Microsoft, computers are not suddenly going to start reading our minds and doing what we want automatically just because everyone in the world has to have a Passport account. No, Sun, we're not going to be able to analyze our corporate sales data "as simply as putting a DVD into your home theatre system."

Remember that the architecture people are solving problems that they think they can solve, not problems which are

It's nice that we can use XML now for the format on the wire. Whoopee. But that's about as interesting to me as learning that my supermarket uses trucks to get things from the warehouse. Yawn. Mangos, that's interesting. Tell me something new that I can do that I couldn't do before, O Astronauts, or stay up there in space and don't waste any more of my time.


Feedback on Programmer Compensation

Monday, August 28, 2000

Here's some feedback from my original programmer compensation article. Based on this feedback, I rewrote the policy: the new one is here.

Your level 13 and 14 seem a little odd, especially 14. Firstly, creating new programming languages doesn't sound like a very good indicator of exceptional quality or intelligence, though I think I know what you were intending. For instance, you and I clearly know a great many very bright people. How many general purpose languages are there?

Also - the 10 years plus fits a great many people - like myself. If anything, those are the people with the CTO-like skills. Level 13, CTO-like skills with 5-10 years of experience...well, that's partly why so many "dot coms" are "dot going away."

Personally, I like performance-oriented compensation. And companies that are slow to hire and quick to fire stand a better chance of insuring they have credible, reliable technology staffs.

I'll probably have some more useful comments later.

- David Geller

Neat. I like the egalitarian and transparent nature of it. If I was still coding, I'd mail you a resume right now.

Of course, you've removed almost all individual incentives, which may or may not be ok.

Might I suggest a radical approach when it comes to the stock options? Silicon Valley craziness has made granting stock de rigeur, but it doesn't really accomplish what it's meant to. Usually, only founders and top management own enough stock to make them care deeply about the company's fate and problems. With later-term employees, they own a bit to make them feel franchised, and can be incented through promises of more.

Unfortunately, it's with the early employees that stock options tend to fail. They have too much for subsequent grants to make much of a difference, and too little for them to be committed to the same extent that management or founders are.

Also, if you mis-plot the risk curve that determines how quickly option grants scale down, there's almost nothing that you can do to rectify the situation later (short of granting a boatload of options to the later people and destroying your stock structure forever).

I'd suggest a different approach, that doesn't apply forever - profit sharing. You can share a very large percentage of the company's profits when the company is young, and slowly bring the percentage down over time as the company matures. At the end of the day, it amounts to the same thing, as the discounted profit stream is the equity value of the company. However, you avoid irrational temporary market swings, and more importantly, if you make an allocation mistake, you're only stuck with it until next year, not forever.

The situation that you're trying to avoid is having someone mediocre but early sitting on a ton of options whereas someone who has joined much later but is doing a great job has far less stock. Best intentions aside, it happens ALL of the time, unless you've been a genius about keeping the risk / reward ratio sensible.

Of course, options have positive tax benefits over profit sharing, but you get my drift.

- Naval Ravikant

No - don't hire consultants - you are definitely on the right track. The only thing that worries me a bit is the 'years of experience' yardstick although the fact that you are linking it to technology exposure is good. Exposure and experience do not necessarily make a better programmer but they certainly do help as long as the person has the extra abilities to be a lateral thinker and a 'solutions' person.

People who do/try different things could also have the edge over those who exclusively program - for instance, I reckon that my 10+ years as an analyst programmer working on different systems certainly helped me be a better dba.

Also be wary of the person who has had superficial exposure to heaps without being expert in any - they abound.

I am sure that you know where you are going with this and will succeed admirably. You should know your market and what you need to pay/offer to get the right people.

- Dale Goopy

I think your level system, the bonuses (as percentage of income), and the stock option scheme are quite solid. Your levels seem to be well thought-out: I can think of few people who would not fit either within one category or between two categories.

However: your startup package exhibits a flaw. By inflating the appearance of compensation rather than actual compensation, you draw certain types of individuals, don't you think? Consider your case of the new college hire. She (to pick a random gender) is offered a position for $70k / year by a competitor. You counter with $60k / year, plus $15k startup. "Woo-hoo!", she thinks, "I'm rich! $5k extra!" After two years, howver, the competing job would have grossed her $140k while yours grossed her $135k, and this delta increases by 10k / year. She is probably one of three types of people, it would seem:

  1. She can't do elementary math.
  2. She is more interested in the short term (< 24 months) than any sort of long term.
  3. She plans to grab you for the cash and ditch you after a year.

Do you really want any of these people working for your company?

Also, your equality of salary increases the granularity of your system. This generates a stair-step: an employee knows not to expect a raise (at least one unequal to that your coworkers receive) until he or she crosses a certain threshhold. It leaves the employee in the situation of saying "I made level 13 eighteen months ago, so I probably have (on the average) three and a half years left until my next significant pay increase." This person may foresee greener pastures, methinks.

On the whole, I think this plan has significant merits. I do think it requires some tweaks, however.

- Joshua McGee

You seem to have come up against a difficult question of fairness, particularly when taking on new hires.

I wish I knew the answer to these questions. I think you're right on having an open and fair policy, I tend to believe that an open book is essential for fairness. I'd guess that the issue of the startup bonus incentives offered can then be influenced by the group of programmers you already have (often loosely referred to as 'the team' ;).

If (following your example) the potential new hire understands your system, but goes on to take the other position anyway, at least they may be tempted back after they're in place & get over the honeymoon period. (not that you'd want to make this a habit of the hire ;).

I think one of the big things that's missing from many organisations is that when someone accepts a position, there's a whole plethora of lifestyle issues to be covered. Where the applicant can see that the new workplace welcomes their lifestyle, and offers a balanced merge between the lifestyle and work, then you're bound to have them being settled in working productively sooner.

- Ray Goopy

Hello Joel, excellent site BTW! Great articles, I wish every place had your ethics.

I think the salary ranges should be based on the market and location. For instance, I live where 40k a year is a nice salary. I have been recently offered a job in San Francisco for 74k but up there that isn't very much.

Or, are you basing the salary on your area?

Also, you mention college grad. Does that mean all your levels require some college?

As far as signing bonus, if I were going to re-locate a signing bonus is very important.

Just thought I would provide some input.

- Scott Burton

I don't think I agree with giving everyone in a particular level the same salary. No two people have the same contribution towards a company's success, so why should their salaries neccessarily coincide? I feel that this policy will simply frustrate the more productive members of your staff.

- Sid

Some comments:

  • I found it interesting that I didn't fit anywhere in your chart, with 23 years of programming behind me. Some of us have spent more time teaching and mentoring than writing new computer languages. IMO, that's a good thing.

  • You've left out one important part of programmer compensation, in my experience. Give every programmer a conference and training budget. Geeks want to go hang out with other geeks. Let them decide if they want to blow the budget on a Geek Cruise, attend JavaOne, or stay at home and buy a copy of every ORA book ever published.

  • I'm also a fan of a tools & toys budget. Let each geek decide themselves which is more important: a new nerf gun, a fancy trackball, or a better set of speakers? Each can be an major productivity booster for the right person.

But then, after 13 years in the corporate world, I've found I'm much much happier being freelance, so what do I know? <g>

- Dori Smith

six different levels, Level 9 through Level 14

I assume you got these levels from somewhere else you've worked.

Why not just cut through the obfuscation and make 'em 0-5 (or 1-6 if you feel bad about someone being a zero)? If you're keeping slots open for administrative, QA, and janitorial types, why not just put 'em in another class entirely?

You've got interns on the list, and you've got team leaders, but you have no spot for a senior programmer who has no desire to lead a team. There are lone-wolves like this who simply don't work and play well with others. Maybe you're not interested in hiring them, but if you are, you should account for them somewhere.

You also don't seem to have a place in your matrix for a teacher. By that I mean a mid-level to senior programmer whose strength is being able to teach others (including more senior people) skills they don't already have. Especially if you're planning on hiring interns at any point, you'll need this sort of person. A good teacher may not be extremely productive in developing new code.

Another one you've omitted is the professional maintenance programmer. The person who comes in and can debug a complete dog's dinner of code. The person who can see that there are three ways to fix code: 1) massive rewrite; 2) minor rewrite of a function or two; or 3) one-line change with a heavy comment about how someone might wish to rewrite the code later. There are very good reasons for each of the approaches, and knowing which to pick isn't something everyone is good at. For that matter, some folks are much better at debugging, while others are born architects. The skill sets needed for the two positions are very different. How do you reconcile the two? If you're being hired guns, it's quite possible the person with better debugging skills will actually bring in more revenue to the company. I've seen this overlap with the teacher position more frequently than other types of programmer.

Compensation consists of:

  1. Salary
  2. Annual bonus
  3. Benefits
  4. Stock options
  5. Startup bonus

You left out time off (though you may consider it part of benefits). The reason I'm running my own business is that no company was willing to hire me on the terms I was looking for. I was looking for a job that was roughly full-time (but in fewer, longer days per week) for about 3/4 of the year. I'd have been willing to take 3/4 (or even slightly less) of full-time salary in order to have the time to enjoy things other than work. In most companies, the only way to do that is to work a year or two, and then either take a leave of absence or quit, knowing you'll have to look for a new job when you return.

Benefits are equal for all full-time employees.

I think a better solution is to have a shopping-bag concept of benefits. Employees get a certain amount of benefits, and can apply them to different types of benefits. A single employee will spend less on health-insurance, but may want to spend more on the health-club or a tuition-reimbursement program. A married person with children will need to spend more on health insurance and day care, but may not have time for the health-club. There are a lot of variations, and one size does not fit all.

One thing I've been pondering is simply punting on fixed benefits and saying "You have n% of your salary to use on benefits. You can allocate it among the following: health-insurance, dental insurance, 401K, ESPP, vacation time, discounted hardware purchases, tuition reimbursement, valet service, tax preparation, etc..." That can be tricky tax-wise, but it's even trickier for the employee to figure some of that stuff out. Spend a few extra $$ to have the corporate accountant figure out how to make it work, and save the employees the hassle.

Stock Options....

Feh! on stock options. They interest me little if at all. If the company is publicly traded, a good ESPP is more interesting. If the company is not publicly traded, stock options are worthless scraps of paper, and may actually cost me money in taxes for a net-negative. Of course the company may go public and do well, but it also may fold before reaching IPO.

And if we're having trouble hiring because of low salaries, we'll raise the salaries for everyone, even the people we've already hired.

One way of looking at salaries is asking yourself: "What would I have to pay to contract this programmer back six months from now after she leaves because we don't pay her enough now?"

The answer to that question can be pretty illuminating. One company I quit was simply unwilling to pay to contract me back after I left, even though they've had two problems I could have walked in and fixed rapidly. Instead, they wanted the security of having someone on-staff who had spent the time to figure it out. Good for them.

Another company that wouldn't meet my desired salary has since contracted me for almost six months worth of that salary for two months work. I suspect they're regretting their decision, but they do have more flexibility now.

as a company, we are very concerned with equity and fairness in salaries. We think it will be extremely valuable to maintain salaries, bonuses, and benefits for programmers at equal levels rather than negotiating individually with everyone and then having people upset that they are not getting a "fair" salary

I think if you truly want to be equitable, you're going to find out that there will be enough special cases, that your pay-parity will quickly evaporate. I've mentioned a few, but there are more that I'm sure I haven't thought of. Being fair is a good ideal, but because of the special cases, equal pay is unlikely to be fair.

- Dave Polaschek

In the company I work for, we do negotiate salaries, which I agree with you it is wrong. But I have one problem in your scheme. Let's say you have 2 developers 1 year experience, same exact level, but one is amazingly smart and fast, the other one is slow, fast and slow means time to finish a certain task in a certain amount of time without major bugs, etc.. etc..., how can you differentiate between those, in this scheme you can't because they will take the same salary and the same bonuses, I see that the motivation to excel is not there... I agree with the same salary scheme for the level, and the startup bonus, but the drive for each person to excel is important too. This might be achieved by an indvidiual bonus or other ideas... What do you think ?

- Ahmed Badr

Editor's note: I don't really believe in individual bonuses or performance incentives, because I've heard a lot of research that says they don't work:

Incentive Pay Considered Harmful

But, to answer your question more specifically, if I really had one programmer who was simply much more productive than usual and wrote code much faster, we would probably just bump them up a level even though they don't meet the other criteria. I don't want the criteria to be hard-and-fast, just general guidelines. - Joel

I think that you're totally on the right track, and identifying your levels as mere guidelines is key. I would give the levels names, fun or serious. Numbers make me think of government pay scales, and leave less flexibility to change them later (unless you like decimals or re-classifying everyone).

I do have to disagree with using large startup bonuses to bridge an offer gap. It creates a mercenary ethic from the start. If a salary of X doesn't make me happy now, as an "unknown risk" to your team, I'm going to be less happy with it down the road once my value to the team has been well-established. I would cap the total startup bonus at $10k and extol the value of the benefits package. Vacation time to me is worth at least twice it's straight salary value, and it's a benefit that most American companies (esp. tech!) are stingy and inflexible with.

(I'm working with a guy who wasn't able to use all of his vacation time last year, a serious no-no in Germany, so he's wound up with 8 weeks for this year. Being on our American payroll, I have serious vacation envy! Not enough for me to switch payrolls, but you can be sure that it will be the only compensation matter discussed at my next review ;-)

- Bryce

Look at Steve McConnell's stuff at construx.com. He has various programmer levels etc. etc. Looks like it's similar to what you've already shown, so maybe you've already been there. We're a pretty small shop with 25 or so programmers and recently went to a three-level basic structure: engineer, Sr. Engineer, Principal Eng. Early feedback is that's not enough granularity. We have a pretty wide range of skills in each level. Probably go to more like you're thinking in next go round.

- Chris Markle

Editor's note: Construx's stuff is exactly what I was trying to reconstruct. Steve got the idea and copied the ladder from Microsoft, and I did do, but I was trying to reconstruct it from memory! - Joel

I like the idea of having levels and paying everyone according to their level. Publishing uses this system for the first two levels (editorial assistant and assistant editor) and it creates a lot of camaraderie since everyone is getting paid the same. Once you're an editor, though, your salary is based on your years of experience and the P&L's of your books. You may want to consider using the level system for your more junior employees but having more freedom to distinguish between the senior ones.

You don't mention it specifically, but I'm assuming that it will be public knowledge what everyone's level is and how to progress to the next one. This is a key factor. I like the consenus approach about setting levels, so long as the employee gets a chance to present their side of it.

What about annual raises? I think everyone expects some sort of annual raise even if you can still hire new programmers at the same wage level as the old ones. And honestly, someone who's worked for you for a year is worth some degree more than someone with equal skills who doesn't know anything about your company or your software.

- Jill McFarlane

On today's topic of compensation, I really like the idea of open and fair salary guidelines. I've never truely understood why salary is such a taboo topic when we'll tell everyone how much we paid for a car or house.

Perhaps it is a bit of utopian, but one would hope that if an employer adopts an open and fair salary policy, they would fully grasp being fair and open and a corporate policy.

- David Benson

How about individual bonuses awarded directly by the employees? I can imagine a bonus plan wherein every employee is given 5% of their own salary to award to other employees. Does incentive compensation of this sort also have the same problem as other incentive compensation?

- Kevin Postlewaite

I'd like to advise you to go ahead with your compensation ideas based on my experience with companies that have done things that way. Sadly, I can't. Every company I've been in has kept these things secret. Instead I urge you to do it simply because I'd like to work at a company with those policies.

I work for Dictaphone (private), which was recently bought by Lernhout & Hauspie (public), so we're getting stock options soon, but it's supposed to be a big secret how the amounts are awarded. We got mail with our share amounts saying do not discuss this with coworkers! What's up with that? Amazingly, many members of my team went along with it! Fortunately a few of us did an "I'll show you mine if you show me yours" session, and we were able to figure out that it's simply based on salary grade. Big deal. Why not just publish that? Secrets do more harm than good.

I suppose that there will always be resentments and perceived injustices with compensation policies. You have to accept that. But keeping it open and telling everybody up front that it's open will, I believe, keep it from interfering with the work.

- John Sands

Earlier this year I was handed a tremendous challenge - to pull together a disorganized department and then design, implement, QA and ship a terrifically complex system with a do or die release date ... of three months later. I can tell you all about it some time because it worked and I learnt an enormous amount about development, management and design but what I wanted to mention here are two things the CEO did that were strong factors in the success of the project - both rather irregular.

#1 - Personal pressure and help: At the time, there were two levels of management in between myself and the CEO. Both were dysfunctional. The CEO cared more about the project than about "dis"ing the two middle managers and set up a weekly status meeting between himself and me to go over the status of the project and do risk assessment. No one wants to disappoint his/her CEO, thus, boy was I on top of the project. Also, if I was having trouble with resource issues he could cut through any red tape and make things happen for me.

#2 (relevant to the bonus question) - at the first status meeting he gave me a massive bonus (options that are currently worth about $150,000) and said that he likes to give bonuses BEFORE the success - he finds it's much more concrete motivation. Now this is a bit of a dirty trick but it sure does work. I plan to try it with my team next real crunch.

I really like your approach to salaries, though you need to have a certain amount of flexibility with the ranking. One of the biggest sources of tension we have in the department is due to people think other people are getting paid more than them. One of the biggest headaches we have when hiring is dealing with people who are good but would throw our salary scale of kilter. Your suggestion solves both very elegantly. The one question I have is what do you do with salary reviews? Over here, people expect a 10-20% raise every year (in addition to any raise due from a "rank" promotion). Would you only give people a raise if the raise in rank? Or maybe instead of a salary raise have an individual bonus at the time of the salary review that would do the same as the start up bonus?

- Anonymous


Finding an Office in New York City

Friday, March 28, 2003

This may be, possibly, the most off-topic article I've ever done here. But I try to write only about things I know about, and, recently, I learned a lot more about commercial real estate than I ever imagined I would need to know.

My company,

The dismal stock performance probably came from the fact that when you relocate more than a couple of miles, some employees' lives would be too disrupted to make the move, so you lose a lot of employees, and all the institutional knowledge, skill, and experience that comes with those employees. While I was working at Viacom one of their companies, Blockbuster, decided to move from Florida to Texas after they hired a new CEO who lived in — Texas! What a coincidence! Only a small portion of the employees made the move. For years and years the business press watched agog as Blockbuster made mistake after inexcusable mistake, re-trying all kinds of ideas that had failed only two years earlier.

But I digress. We're not losing any employees and we're just moving across town, to a location that is an easy walk from every subway line in NYC and three mega commuter hubs, bringing us within easy commuting distance for 17,000,000 potential employees.

Temporary Space and Incubators

Step one in finding an office in New York: allow nine months. I am not kidding about this. It will take about two months to find the space, a month to negotiate, two months to sign the lease, a month for your architect to design the build out and three months for construction. If you run out of time on your old lease, chances are (look closely) you have a clause in there which doubles or triples your rent if you stay past the end. This is called the overhang and next time you negotiate a lease, you'll remember to negotiate a lower overhang.

Nine months. You may think you can shorten that by taking pre-built space. The problem: there's hardly anything pre-built in the office market in New York. Almost everything you see is going to be either completely raw, or it's going to be a weird warren of dark rabbit offices with inexplicable bars between certain rooms, filthy carpet, and disgusting yellow-stained acoustic tiles.

What there is, if you have to move tomorrow, is temporary executive suites. These are the things you see advertised in the newspaper that say "Move Tomorrow! Plans from $100 a month!" What the ad doesn't say is that the $100 a month is for a mailbox only, with everything else a la carte. If you really want an office in one of these executive suites you're going to pay through the nose: about five to ten times as much as comparable office space. It's like living in a hotel instead of living in an apartment, or flying Concorde to London. Yeah, it's convenient, but is it really $10,000 a month convenient? Still, all the executive suites in the city are full, usually an honor roll of venture-capital backed companies working hard on keeping their cash burn up to respectable levels to impress the investors. This is because normal landlords hate venture capital funded companies. The few landlords in New York that rented to VC startups during the dotcom boom have learned never to do that again. Here's why: venture funded companies only have enough cash to last a few months or at most a year, after which they have to raise another round. If they don't raise the round, they close. Probably 80% of these companies close within a year. The way most leases are written, a landlord literally loses money if the tenant leaves after a year. If you're venture funded, and your VC won't sign a five year lease for you, you may be stuck paying the Rich Tax, paying $200 per foot (per year) to be in executive suites when the market rent is $20 - $30.

Another option is the so-called incubators. In New York City, we have TechSpace, EEmerge, Kickstart, and probably a few others. I am sort of surprised these places survived the dotcom bust, but they did, and they've lost their arrogance. They no longer act like VCs, they don't care about your business plan any more, and they don't want 5% of your pathetic worthless stock, they just want to get paid. They won't even kick you out after 3 months any more.

Incubators are a terrible place for programmers, because they took the baby chick metaphor a bit too far. The scientific geniuses that built these places actually think that the closer you shove people together the warmer they will be and thus the faster they will grow. What you get at the incubators is a room the size of your bedroom with desks around the outer wall where you can cram 16 people, all for about three thousand bucks a month. Oh, you wanted a window? Make it four thousand. Incubators are a smidgen cheaper, per square foot, than the executive office space, but they are just not an acceptable long-term solution.

Normal Office Space

A quick survey of normal office space.

First, it's usually not built, and if it is, somebody has probably been using it. If the walls are configured to your liking, you may be able to paint and recarpet it (about $7 per square foot) and move in. Otherwise you're going to have to do construction. Plan on this costing $50 per square foot. So if it's a 3000 square foot office, expect to spend $150,000 on the construction. I know, this sounds like a lot. While you are looking for offices, brokers will tell you that you can construct an office for less than $50 a square foot. They are lying to make the deal. Do not fall for it. The only way to pay less is if a lot of the work is already done (on our office, the demolition, ceiling, sprinklers, air conditioner, bathrooms, and finished floors were all done, so we're hoping to spend $25/ft on construction.)

In today's market, you can usually get the landlord to pay for some of this, probably $10 to $25 a foot. They will want receipts and stuff and they will only want to pay for walls and air conditioning ducts, not Aerons and plasma TV screens. Some landlords will also offer to do the construction themselves for you. If you take them up on it, you'll get the cheapest possible construction and it won't be very nicely designed, so you may be better off hiring your own architect and contractor.

In the office space market in the US, offices are categorized as Class A, B, or C. Class A means new, top-of-the-line construction. In New York, that means one thing only: modern steel and glass high rises. Anything else is not Class A. A classic old skyscraper from 1913 designed by Cass Gilbert with incredible detailing, beautiful gargoyles, a complete renovation from top to bottom and everything is spotless and new is still not a Class A building, even if it's nicer than most Class A buildings.

Class B and Class C don't mean anything, there's no formal definition, so nobody bothers to use these terms in the city any more. There's Class A and there's Everything Else.

For the purposes of high tech startups, unless you need a whole floor of a skyscraper, forget about Class A. It's not that you can't afford it, it's that they are not going to subdivide floors for you. These buildings are leased 10 floors at a time.

That leaves you with a wide variety of old buildings, often loft buildings, in various stages of renovation. The best ones have been recently renovated, the elevators are new, everything is spick-and-span and the lobby is nice if not glorious. The worst ones have a spanish deli as the lobby, the (one) elevator is manual and operated by a scary old man, and there are still a lot of clothing factories in the building operating heavy machinery that goes bonk bonk bonk all day long. Almost all of these buildings are family owned, although they might be managed by one of the large real estate management companies like Newmark or GVA Williams.

What If I Really, Really Don't Want to Build?

There are a few buildings which cater to small tenants where the offices are basically built out and at most you'll need paint and carpet. The Greybar Building near Grand Central, the Fisk Building near Columbus Circle, and the hugely unpopular Empire State Building. Greybar and Fisk always have spaces available on short notice at "market rents," about 50% - 100% more than you'll pay for raw space in the Garment District or Wall Street but reasonable for their neighborhoods. Greybar is a funny ghost town building, right on top of Grand Central; it's full of offices of just-retired lawyers who live in Connecticut and can't quite let go, so they keep an office in the Greybar building, but then they never really go there.

Do You Need A Broker?

In the commercial market, brokers are paid by the landlord. Usually something like 6% of the total value of the lease over its entire term. This is a lot of money for a little bit of work. So as soon as you let it be known that you are looking for office space, you will have brokers falling all over you.

You have the option of using a tenant's broker. A good tenant's broker does all the legwork for you, looking at lots of spaces to find one and then showing you only the best options to choose from. The trouble is, there are hardly any good tenant's brokers. Most of them are just too lazy. The will tell you to meet them at Starbucks at 10:00 am, and they'll bring a printout they made at 9:45 that you could have gotten online (more about that in a moment), and they'll make you walk around with them while they get their act together and discover that these spaces don't really exist.

Tenant's brokers will tell you that they have the inside track on office space that isn't listed, especially sublets. The best ones will, but only the very best ones. Most of them are just making the same printouts you can get yourself.

Bottom line: you don't really need a tenant's broker to show you spaces. OK, the other thing the tenant's brokers will tell you is that they represent you, not the landlord, so they will have your interests in mind. This is vaguely true, but they're still getting paid by the landlord, and they still make more money if you overpay, so it's hard to trust them. If you're worried about landlord sharks taking advantage of you, stop worrying about that. Why? Because you'll have a good lawyer on your side when you sign the lease. You need a lawyer to review your lease anyway, that's your real protection from the landlord, not a broker who gets paid by landlords.

Are there disadvantages to having a tenant's broker? There are two. First, you may have to sign an exclusive. In today's market, with vacancies skyrocketing and no new tenants, you should flatly refuse. There's no reason to sign an exclusive. Just promise the broker that if you take a space which they showed you first, you will see to it that they get their commission.

The other disadvantage is that if a tenant's broker finds you a space and you take it, they will be splitting the commission between the tenant's broker and the landlord's broker. That means the landlord's broker only gets half the commission. That means, all else being equal, that the landlord's broker would rather do those deals where they get the whole commission, when they have a choice. That means that if you're competing for space with another potential tenant, and that tenant has his own broker and you don't, then the landlord's broker is going to like you much more... twice as much, in fact. Although theoretically it's up to the owner of the building to decide whom to rent to, you can bet that the landlord's broker will find ways to make the owner like you more.

How Do I Find a Space Without a Broker?

Easy as pie.

First, do searches on two websites, which overlap some but not entirely in their listings: CityFeet and MrOfficeSpace.

Next, check the Sunday New York Times. (Not online. The online version just redirects you to CityFeet, which has a superset of the print classifieds). Circle everything that looks appropriate.

If you want cheaper, artsy space, check the Village Voice.

Avoid two things: ads that sound like executive office spaces, and ads that are just pitches for brokers disguised as listings. You can identify these because they don't give any details about the space, like the address. If the address is not in the listing, don't bother responding, it's almost certainly a broker who is fishing for potential tenants. He'll tell you to meet him at Starbucks at 10:00 am where he'll show up with exactly the same list you printed out from MrOfficeSpace.

Next step: do not call the broker in the listing. Yes, they can show you the space, but you don't need them yet. First go to the building in the listing and see if you like the building. Check out the lobby. Read the list of tenants to see if they're mostly software companies, architects, and graphic designers, or if they're mostly clothing factories, importers, and methadone clinics. Check out the neighborhood. In the bad buildings, people get into the elevators talking to themselves. In the good buildings, people appear to get into the elevators talking to themselves, but they have a tiny earphone, so it's totally different, they're just on the phone.

Like the building? Here's the next step, for which you should be reasonably well dressed. Go into the building's service entrance and ask to speak to the super. Tell the super you're looking for office space and you want to see the space in his building. He will show it to you. In better buildings, he may ask for a business card. That's all there is to it.

Now, if you like the space, call the broker.

"What would have happened," you may ask, "if I had just called the number in the ad?" Well, nothing terrible. The broker would have met you in the building lobby, and then he would have asked to see the super, and the super would have shown you the space. You see, the super has the keys and knows how to drive that cool manual elevator that goes to the floors where the fancy automatic elevators don't stop because there are no tenants there.

At this point, if you don't like the space, the broker will offer to show you a zillion other spaces. If he's smart he'll have the MrOfficeSpace printout that you also have. It doesn't hurt to walk around with him and look at spaces, he may know the neighborhood and be able to give you the gossip. It's worth spending some time talking to all the brokers you meet. Ask them what buildings have nice owners, and what buildings have nasty owners. They will lie a little bit about the buildings in which they have an interest, but if you keep asking and average the results you'll eventually learn the reputations of the landlords in the neighborhood.

How Do I Know If I Like The Space?

I kept looking at spaces, not particularly knowing if I liked anything or not, and one nice broker said to me: "Don't worry. When you find the right place for you, you'll know it." And he was right. The next day I walked into a place that was beautiful: natural light on three sides, a great renovated building, beautiful new wood floors; instead of taking notes I just wrote "incredibobble" in my notebook. That's the place you want. If it doesn't happen keep looking.

Gotchas to watch out for:

  • If there are windows, figure out what might happen that would block your views. If the building is next to an empty lot or a "taxpayer" (one story building), as soon as the economy gets better, a building might go up there and it might block your view.
  • Check out the bathrooms. Some buildings have really gross bathrooms and people forget to look at them.
  • Sit quietly in the space for a few minutes and listen to all the noises. Are there factories nearby? Noisy people upstairs banging around? A noise you don't notice when you're walking around talking to a broker may be torture when you're trying to concentrate on writing code.
  • Make sure the building has 24/7 access. Some buildings in New York actually close.
  • How are you going to get air conditioning? Is there something in place or will you have to install it? When you install it will it block your only window?
  • Are there enough elevators? Older buildings with only one or two elevators can mean really long, irritating waits.

How Big A Space Do I Need?

Space is measured in RSF: rentable square feet. This is a number that bears only a fleeting relationship to the actual size of the space. I have seen spaces that are 1000 RSF which are the same size as spaces that are 3000 RSF.

How could this be, you ask?

Well, you don't want to know. Officially you get RSF by taking the total size of the space and multiplying it by the building's loss factor. That's a made up number which they multiply the size of the space by, just because they can get away with it. It's supposed to account for all the shared areas, like the basement, the elevator shafts (counted once for each floor of course), the exterior walls, the interior walls, the bathrooms, the lobby, the Starbucks next to the lobby, the staircases, the fire escapes, and the owner's house in Queens including pool and garden.

Meaning — RSF is pretty useless. When they quote a price, it will always be quoted in dollars per square foot per year. Multiply by the size and divide by 12 to get your monthly rent, and this is the only figure you care about. Sign a lease based on the cost, not the square feet, or they'll "recalculate" their loss factor the next year and raise your rent.

This makes it sort of hard to search for office space, and it's one reason you have to look at lots of spaces (and bring a tape measure.) If you calculated that you're going to need about 2000 square feet, look for places advertised between 1000 and 3000.

Hidden Charges

Find out about all the charges you're going to have to pay. There are a couple of ways New York buildings especially like to stick it to you.

Some buildings don't have separate electric meters for each space. Since they don't know how much electricity you're using, they just have to charge you based on the assumption that you are operating a 100,000 watt radio station with Times Square lighting. They'll charge you something like $3 per square foot per year which is way more than you would spend if you had your own meter.

Some buildings make up things and put them in the lease. "Tenant shall pay $200 per month to maintain the decorative steel trim on the building," my lease said. Hmm. Damn building doesn't even have a steel trim. I made them take it out.

Escalations are another big issue. If the rent is x for the first year, it's going to be x times the escalation in the second year, and so on. This amount should be agreed upon and not based on some weird fabricated thing like "tenant's prorated share of the owner's increased expenses from operating the building." I don't care about the owner's expenses and I'm not paying for the owner to pass along the cost of his 2003 Buick LeSabre to me. Some buildings have a uniquely Manhattan corruption called "Porter's Wage." This means that every year, when they renegotiate the contract with the Porter's Union, the percentage that those salaries goes up by is the percentage escalation that the tenants pay in rent. The theory was that a porter's wage was correlated with the overall inflation in the cost of operating a building, but in reality buildings don't have porters any more, and for some reason the porters they don't have would get really huge salaries, if they had jobs, but they don't, and the tenants are still paying for it.

Another thing the landlord will want to pass on to you is your share of the increase in property taxes. This is reasonably standard. They will even bill you for any money they spent "lobbying" (paying bribes) to get their property taxes reduced. Slimy but common.

How Do I Negotiate The Lease?

Theoretically, everything is negotiable. Practically, you're going to negotiate these big items:

  • monthly rent.
  • months of free rent at the beginning (these days it's not uncommon to get one month for each year of the lease, and even a couple more months while construction is going on)
  • landlord's contribution to buildout
  • length of the lease

For the purpose of negotiations, there are a few things you can pretend to care about which you can later swap to get a better deal:

  • pretend that you want 100% of the profit from a sublet. They won't give it to you, they'll give you 50%, which is actually better for you because it gives them an incentive to allow the sublet in the first place, and you can swap this for something else.
  • pretend that you don't want to pay a security deposit.
  • pretend that you want zero overhang, so your rent doesn't go up if you overstay your lease.

You may think you want to sign as short a lease as possible, because it "leaves your options open." Actually you're probably better off with a longer lease, for lots of reasons.

  • Landlords like long leases, so they'll like you more if you sign a long lease, and you can get a better deal.
  • Rents are super low now. If rents go up, your lease becomes worth cash money.
  • If you grow out of the space, the landlord will let you out of the lease if you take another space elsewhere from the same landlord.
  • If you go bankrupt or close, the landlord is not really going to be able to get anything out of you. If you're incorporated and the corporation signs the lease, you're not personally liable. At worst, the landlord will ask the company's officers to personally sign a good guy guarantee. This says that you will be personally responsible for the rent but only until the premises are vacated.
  • If you need less space, you can probably sublet. You might lose some money on the sublet but you'll probably make money. The landlord will help himself to half of your net profits from subletting.
  • You'll get more free months at the beginning with a longer lease.
  • You'll be able to negotiate more money for construction with a longer lease.
  • The broker will get more money, so he'll be more eager to make the deal happen.

All in all, there's no reason to be unduly scared of long leases. You can start by asking for a three or five year lease, agree on the months of free rent and construction money, and then ask them how much more free rent and money they'll give you if you take a seven year lease. (Logically, this should be proportional — doubling the length of the lease should double the free stuff).

There are thousands of good books on negotiating in general and even a bunch of books on negotiating leases. Read two or three if you've never done this before. Never, ever go into negotiations unless you have a good alternative. You should have two or three options you like and want before you try to negotiate.

After you negotiate the big items, get a proposed lease from the landlord and let your lawyer negotiate all the little details to his lawyerly heart's content.

Do I Need a Lawyer?

Absolutely. First of all, the errors-and-omissions insurance you carry (you do have errors and omissions insurance, right?) requires you to have a lawyer review any contracts you sign. But it's a great idea anyway.

I found a lawyer who works for another landlord. He spends most of his days in court trying to evict tenants. I thought he would be perfect to protect me from the likes of — him. And he was. As he was reading through a lease, he said to me, "OK, this paragraph, I really like this clause, I think I'm going to steal it and use it in my own contracts." Pause. He reaches for a red pen and crosses out the paragraph. "But you're not signing it." This guy was great. He knew that the standard phrase in the lease where it talks about leaving the space in "broom clean condition" was a classic way to avoid returning the deposit. "Sure, it was clean," the owner says, "but it wasn't broom clean. Prove to me that it was broom clean." We crossed out "broom" and wrote "reasonably." You Need A Good Lawyer For This.

Here's another reason you need a lawyer. The lawyer can say to the landlord, "Whaddaya mean if we're 3 days late on the rent we get evicted? We want the right to be 15 days late!" When your lawyer says this, no offense is taken. He's just the lawyer. But if you, the tenant, try to say it, the landlord will say, "What, are you planning to be late on the rent?"

Oh, It's All Too Hard!

I know. I know. And it has nothing to do with writing software. But as soon as you sign the lease, you can hire an architect and start talking about the fun stuff, like where the pool table goes.

Other Links

TenantWise has a lot of great articles about all this stuff, although you have to register to read it all, it's worth it.

Crains New York Business covers the commercial real estate beat like no other publication. Read it for a few months before you need office space, especially pay attention to the boring stories about deals, so you know what the market is like and negotiate from a position of knowledge.

Read the last few months of news at CityFeet, too.

Discuss your experience on the Joel on Software New Yorker's Forum.


Fire And Motion

Sunday, January 06, 2002

Sometimes I just can't get anything done.

Sure, I come into the office, putter around, check my email every ten seconds, read the web, even do a few brainless tasks like paying the American Express bill. But getting back into the flow of writing code just doesn't happen.

These bouts of unproductiveness usually last for a day or two. But there have been times in my career as a developer when I went for weeks at a time without being able to get anything done. As they say, I'm not in flow. I'm not in the zone. I'm not anywhere.

Everybody has mood swings; for some people they are mild, for others, they can be more pronounced or even dysfunctional. And the unproductive periods do seem to correlate somewhat with gloomier moods.

It makes me think of those researchers who say that basically people can't control what they eat, so any attempt to diet is bound to be short term and they will always yoyo back to their natural weight. Maybe as a software developer I really can't control when I'm productive, and I just have to take the slow times with the fast times and hope that they average out to enough lines of code to make me employable.

What drives me crazy is that ever since my first job I've realized that as a developer, I usually average about two or three hours a day of productive coding. When I had a summer internship at Microsoft, a fellow intern told me he was actually only going into work from 12 to 5 every day. Five hours, minus lunch, and his team loved him because he still managed to get a lot more done than average. I've found the same thing to be true. I feel a little bit guilty when I see how hard everybody else seems to be working, and I get about two or three quality hours in a day, and still I've always been one of the most productive members of the team. That's probably why when Peopleware and XP insist on eliminating overtime and working strictly 40 hour weeks, they do so secure in the knowledge that this won't reduce a team's output.

But it's not the days when I "only" get two hours of work done that worry me. It's the days when I can't do anything.

I've thought about this a lot. I tried to remember the time when I got the most work done in my career. It was probably when Microsoft moved me into a beautiful, plush new office with large picture windows overlooking a pretty stone courtyard full of cherry trees in bloom. Everything was clicking. For months I worked nonstop grinding out the detailed specification for Excel Basic -- a monumental ream of paper going into incredible detail covering a gigantic object model and programming environment. I literally never stopped. When I had to go to Boston for MacWorld I took a laptop with me, and documented the Window class sitting on a pleasant terrace at HBS.

Once you get into flow it's not too hard to keep going. Many of my days go like this: (1) get into work (2) check email, read the web, etc. (3) decide that I might as well have lunch before getting to work (4) get back from lunch (5) check email, read the web, etc. (6) finally decide that I've got to get started (7) check email, read the web, etc. (8) decide again that I really have to get started (9) launch the damn editor and (10) write code nonstop until I don't realize that it's already 7:30 pm.

Somewhere between step 8 and step 9 there seems to be a bug, because I can't always make it across that chasm.For me, just getting started is the only hard thing. An object at rest tends to remain at rest. There's something incredible heavy in my brain that is extremely hard to get up to speed, but once it's rolling at full speed, it takes no effort to keep it going. Like a bicycle decked out for a cross-country, self-supported bike trip -- when you first start riding a bike with all that gear, it's hard to believe how much work it takes to get rolling, but once you are rolling, it feels just as easy as riding a bike without any gear.

Maybe this is the key to productivity: just getting started. Maybe when pair programming works it works because when you schedule a pair programming session with your buddy, you force each other to get started.

When I was an Israeli paratrooper a general stopped by to give us a little speech about strategy. In infantry battles, he told us, there is only one strategy: Fire and Motion. You move towards the enemy while firing your weapon. The firing forces him to keep his head down so he can't fire at you. (That's what the soldiers mean when they shout "cover me." It means, "fire at our enemy so he has to duck and can't fire at me while I run across this street, here." It works.) The motion allows you to conquer territory and get closer to your enemy, where your shots are much more likely to hit their target. If you're not moving, the enemy gets to decide what happens, which is not a good thing. If you're not firing, the enemy will fire at you, pinning you down.

I remembered this for a long time. I noticed how almost every kind of military strategy, from air force dogfights to large scale naval maneuvers, is based on the idea of Fire and Motion. It took me another fifteen years to realize that the principle of Fire and Motion is how you get things done in life. You have to move forward a little bit, every day. It doesn't matter if your code is lame and buggy and nobody wants it. If you are moving forward, writing code and fixing bugs constantly, time is on your side. Watch out when your competition fires at you. Do they just want to force you to keep busy reacting to their volleys, so you can't move forward?

Think of the history of data access strategies to come out of Microsoft. ODBC, RDO, DAO, ADO, OLEDB, now ADO.NET - All New! Are these technological imperatives? The result of an incompetent design group that needs to reinvent data access every goddamn year? (That's probably it, actually.) But the end result is just cover fire. The competition has no choice but to spend all their time porting and keeping up, time that they can't spend writing new features. Look closely at the software landscape. The companies that do well are the ones who rely least on big companies and don't have to spend all their cycles catching up and reimplementing and fixing bugs that crop up only on Windows XP. The companies who stumble are the ones who spend too much time reading tea leaves to figure out the future direction of Microsoft. People get worried about .NET and decide to rewrite their whole architecture for .NET because they think they have to. Microsoft is shooting at you, and it's just cover fire so that they can move forward and you can't, because this is how the game is played, Bubby. Are you going to support Hailstorm? SOAP? RDF? Are you supporting it because your customers need it, or because someone is firing at you and you feel like you have to respond? The sales teams of the big companies understand cover fire. They go into their customers and say, "OK, you don't have to buy from us. Buy from the best vendor. But make sure that you get a product that supports (XML / SOAP / CDE / J2EE) because otherwise you'll be Locked In The Trunk." Then when the little companies try to sell into that account, all they hear is obedient CTOs parrotting "Do you have J2EE?" And they have to waste all their time building in J2EE even if it doesn't really make any sales, and gives them no opportunity to distinguish themselves. It's a checkbox feature -- you do it because you need the checkbox saying you have it, but nobody will use it or needs it. And it's cover fire.

Fire and Motion, for small companies like mine, means two things. You have to have time on your side, and you have to move forward every day. Sooner or later you will win. All I managed to do yesterday is improve the color scheme in FogBUGZ just a little bit. That's OK. It's getting better all the time. Every day our software is better and better and we have more and more customers and that's all that matters. Until we're a company the size of Oracle, we don't have to think about grand strategies. We just have to come in every morning and somehow, launch the editor.

Discuss


Five Worlds

Monday, May 06, 2002

Something important is almost never mentioned in all the literature about programming and software development, and as a result we sometimes misunderstand each other.

You're a software developer. Me too. But we may not have the same goals and requirements. In fact there are several different worlds of software development, and different rules apply to different worlds.

You read a book about UML modeling, and nowhere does it say that it doesn't make sense for programming device drivers. Or you read an article saying that "the 20MB runtime [required for .NET] is a NON issue" and it doesn't mention the obvious: if you're trying to write code for a 32KB ROM on a pager it very much is an issue!

I think there are five worlds here, sometimes intersecting, often not. The five are:

  1. Shrinkwrap
  2. Internal
  3. Embedded
  4. Games
  5. Throwaway

When you read the latest book about Extreme Programming, or one of Steve McConnell's excellent books, or Joel on Software, or Software Development magazine, you see a lot of claims about how to do software development, but you hardly ever see any mention of what kind of development they're talking about, which is unfortunate, because sometimes you need to do things differently in different worlds.

Let's go over the categories briefly.

Shrinkwrap is software that needs to be used "in the wild" by a large number of people. It may be actually wrapped in cellophane and sold at CompUSA, or it may be downloaded over the Internet. It may be commercial or shareware or open source or GNU or whatever -- the main point here is software that will be installed and used by thousands or millions of people.

Shrinkwrap has special problems which derive from two special properties:

  • Since it has so many users who often have alternatives, the user interface needs to be easier than average in order to achieve success.
  • Since it runs on so many computers, the code must be unusually resilient to variations between computers. Last week someone emailed me about a bug in CityDesk which only appears in Polish Windows, because of the way that operating system uses Right-Alt to enter special characters. We tested Windows 95, 95OSR2, 98, 98SE, Me, NT 4.0, Win 2000, and Win XP. We tested with IE 5.01, 5.5, or 6.0 installed. We tested US, Spanish, French, Hebrew, and Chinese Windows. But we hadn't quite gotten around to Polish yet.

There are three major variations of shrinkwrap. Open Source software is often developed without anyone getting paid to develop it, which changes the dynamics a lot. For example, things that are not considered "fun" often don't get done in an all-volunteer team, and, as Matthew Thomas points out eloquently, this can hurt usability. Development is much more likely to be geographically dispersed, which results in a radically different quality of team communication. It's rare in the open source world to have a face to face conversation around a whiteboard drawing boxes and arrows, so the kind of design decisions which benefit from drawing boxes and arrows are usually decided poorly on such projects. As a result geographically dispersed teams have done far better at cloning existing software where little or no design is required.

Consultingware is a variant of shrinkwrap which requires so much customization and installation that you need an army of consultants to install it, at outrageous cost. CRM and CMS packages often fall in this category. One gets the feeling that they don't actually do anything, they are just an excuse to get an army of consultants in the door billing at $300/hour. Although consultingware is disguised as shrinkwrap, the high cost of an implementation means this is really more like internal software.

Commercial web based software such as Salesforce.com or even the more garden variety eBay still needs to be easy to use and run on many browsers. Although the developers have the luxury of (at least) some control over the "deployment" environment -- the computers in the data center -- they have to deal with a wide variety of web browsers and a large number of users so I consider this basically a variation of shrinkwrap.

Internal software only has to work in one situation on one company's computers. This makes it a lot easier to develop. You can make lots of assumptions about the environment under which it will run. You can require a particular version of Internet Explorer, or Microsoft Office, or Windows. If you need a graph, let Excel build it for you; everybody in our department has Excel. (But try that with a shrinkwrap package and you eliminate half of your potential customers.)

Here usability is a lower priority, because a limited number of people need to use the software, and they don't have any choice in the matter, and they will just have to deal with it. Speed of development is more important. Because the value of the development effort is spread over only one company, the amount of development resources that can be justified is significantly less. Microsoft can afford to spend $500,000,000 developing an operating system that's only worth about $80 to the average person. But when Detroit Edison develops an energy trading platform, that investment must make sense for a single company. To get a reasonable ROI you can't spend as much as you would on shrinkwrap. So sadly lots of internal software sucks pretty badly.

Embedded Software has the unique property that it goes in a piece of hardware and in almost every case can never be updated. This is a whole different world, here. The quality requirements are much higher, because there are no second chances. You may be dealing with a processor that runs dramatically more slowly than the typical desktop processor, so you may spend a lot of time optimizing. Fast code is more important than elegant code. The input and output devices available to you may be limited. <The GPS system in the car I rented last week had such pathetic I/O that the usability was dismal. Have you ever tried to input an address on one of these things? They displayed a "keyboard" on screen and you had to use the directional arrows to choose letters from five small matrices of 9 letters each. (Follow the link for more illustrations of this UI. The GPS in my own car has a touch screen which makes the UI dramatically better. But I digress).

Games are unique for two reasons. First, the economics of game development are hit-oriented. Some games are hits, many more games are failures, and if you want to make money on game software you recognize this and make sure that you have a portfolio of games so that the blockbuster hit makes up for the losses on the failures. This is more like movies than software.

The bigger issue with the development of games is that there's only one version. Once your users have played through Duke Nukem 3D, they are not going to upgrade to Duke Nukem 3.1D just to get some bug fixes and new weapons. With some exceptions, once somebody has played the game to the end, it's boring to play it again. So games have the same quality requirements as embedded software and an incredible financial imperative to get it right the first time. Shrinkwrap developers have the luxury of knowing that if 1.0 doesn't meet people's needs and doesn't sell, maybe 2.0 will.

Finally Throwaway code is code that you create temporarily solely for the purpose of obtaining something else, which you never need to use again once you obtain that thing. For example, you might write a little shell script that massages an input file that you got into the format you need it for some other purpose, and this is a one time operation.

There are probably other kinds of software development that I'm forgetting.

Here's an important thing to know. Whenever you read one of those books about programming methodologies written by a full time software development guru/consultant, you can rest assured that they are talking about internal, corporate software development. Not shrinkwrapped software, not embedded software, and certainly not games. Why? Because corporations are the people who hire these gurus. They're paying the bill. (Trust me, id software is not about to hire Ed Yourdon to talk about structured analysis.)

Last week Kent Beck made a claim that you don't really need bug tracking databases when you're doing Extreme Programming, because the combination of pair programming (with persistent code review) and test driven development (guaranteeing 100% code coverage of the automated tests) means you hardly ever have bugs. That didn't sound right to me. I looked in our own bug tracking database here at Fog Creek to see what kinds of bugs were keeping it busy.

Lo and behold, I discovered that very few of the bugs in there would have been discovered with pair programming or test driven development. Many of our "bugs" are really what XP calls stories -- basically, just feature requests. We're using the bug tracking system as a way of remembering, prioritizing, and managing all the little improvements and big features we want to implement.

A lot of the other bugs were only discovered after much use in the field. The Polish keyboard thing. There's no way pair programming was going to find that. And logical mistakes that never occurred to us in the way that different features work together. The larger and more complex a program, the more interactions between the features that you don't think about. A particular unlikely sequence of characters ({${?, if you must know) that confuses the lexer. Some ftp servers produce an error when you delete a file that doesn't exist (our ftp server does not complain so this never occurred to us.)

I carefully studied every bug. Out of 106 bugs we fixed for the service pack release of CityDesk, exactly 5 of them could have been prevented through pair programming or test driven design. We actually had more bugs that we knew about and thought weren't important (only to be corrected by our customers!) than bugs that could have been caught by XP methods.

But Kent is right, for other types of development. For most corporate development applications, none of these things would be considered a bug. Program crashes on invalid input? Run it again, and this time watch your {${?'s! And we only have One Kind of FTP server and nobody in the whole company uses Polish Windows.

Most things in software development are the same no matter what kind of project you're working on, but not everything. When somebody tells you about methodology, think about how it applies to the work you're doing. Think about where the person is coming from. Steve McConnell, Steve Maguire, and I all come from a very narrow corner: the world of mass market shrinkwrap spreadsheet applications written in Redmond, Washington. As such we have higher bars for ease of use and lower bars for bugs. Most of the other methodology gurus make their living doing consulting for in house corporate development, and that's what they're talking about. In any case, we should all be able to learn something from each other.


Fixing Venture Capital

Tuesday, June 03, 2003

Many software companies these days are built using some form of venture capital. But the VC industry has been hurting lately. A lot of investments in dotcoms turned out to be spectacular flameouts. As a result, VCs are becoming ever more selective about where to put their money. To get funded these days, it's not enough to be a pet shop on the web. Nope! You have to be a pet shop on the web with 802.11b wireless hotspots, or your business plan is going right in the dumpster.

The formerly secretive world of VC has become a bit more transparent, of late. VCs like Joi Ito, Andrew Anker, David Hornik, and Naval Ravikant have created weblogs which are a great source of insight into their thought process. That dotcom thing resulted in three great books by company founders that look deep inside the process of early stage financing (see footnote). But as I read this stuff, as a founder of a company, I can't help but think that there's something wrong with the VC model as it exists today. Almost every page of these books makes me say, "yep, that's why Fog Creek doesn't want venture capital." There are certain fundamental assumptions about doing business in the VC world that make venture capital a bad fit with entrepreneurship. And since it's the entrepreneurs who create the businesses that the VCs fund, this is a major problem. Here's my perspective on that, from a company founder's point of view.

When people ask me if they should seek venture capital for their software startups, I usually say no. At Fog Creek Software, we have never looked for venture capital. Here's why.

The fundamental reason is that VCs do not have goals that are aligned with the goals of the company founders. This creates a built-in source of stress in the relationship. Specifically, founders would prefer reasonable success with high probability, while VCs are looking for fantastic hit-it-out-of-the-ballpark success with low probability. A VC fund will invest in a lot of startups. They expect about seven of them to fail, two of them to trudge along, and one of them to be The Next Netscape ("TNN"). It's OK if seven fail, because the terms of the deal will be structured so that TNN makes them enough money to make up for all the losers.

Although the real spreadsheets are many megabytes long and quite detailed, this is the VC's calculation:

A | Probability of Success | 10%
B | How rich I would get | $1,000,000,000
C | Expected Return (A x B) | $100,000,000

But founders are much more conservative than that. They are not going to start ten companies in their lifetime, they're going to start, maybe, two. A founder might prefer the following model:

A | Probability of Success | 80%
B | How rich I would get | $100,000,000
C | Expected Return (A x B) | $80,000,000

Even though the second model has a lower expected return, it is vastly preferable to most founders, who can't diversify away the risk, while VCs who invest in dozens of businesses would prefer the first model because it has a greater return. This is just Econ 101; it's the same reason you buy car insurance and Hertz doesn't.

The difference in goals means that VCs are always going to want their companies to do risky things. Oh, sure, they'll deny it, but if they were really looking to do conservative risk-free things, they'd be investing in U.S. Treasuries, not optical networking companies. But as an entrepreneur, you're going to be forced at gunpoint to bet on three cherries again and again and again. You know you're going to lose, but the gunman doesn't care, he's got bets on all the slot machines and one of them is going to pay off big time.

There's nothing controversial here. A VC would say, "that's what VC is for: investing in risky ideas." Fair enough. As long as the entrepreneur wants to take a 10% chance, VC may be the way to go. The trouble here is that the VC is now doing a perverse kind of selection. They are looking for the founders with business ideas where the founders themselves think the idea probably won't work. The end result is that VC money ends up being used in bet-the-farm kind of ways. This kind of recklessness causes companies like WebVan to blow $800,000,000 in a rather desperate attempt to buy a profitable business model. The trouble is that they were going so fast that they didn't have time to learn how to spend money in a way that has a positive return, which is, by definition, what you have to do to be profitable.

Here's my philosophy of company growth. A growing company looks like this:

Oh, wait, I forgot to define the Y axis. Let's assume this curve is my revenues:

There are some other things which grow at roughly the same speed. For example, the number of employees:

And the number of people who have heard of your product, which we'll call "PR":

There's also the "quality of your code" curve, based on the theory that good software takes ten years .

I've drawn these curves moving up at roughly an equal rate. That's not a coincidence. In a small company, you regulate each of these curves so they stay roughly in sync. Why? Because if any two of those curves get out of whack, you have a big problem on your hand?euro;”one that can kill your company. For example:

  1. Revenues grow faster than you can hire employees. Result: customer service is inadequate. Let's tune in to Alex Edelstein over at Cloudmark: ?euro;œ[Cloudmark Sales are] pretty swamped, so they're not getting back properly to everyone.... What's happening here now at Cloudmark is a little like the early days at Netscape when we just had too few people to properly respond to the customer interest.?euro;?
  2. Revenues grow slower than you hire employees. Result: you burn cash at a ridiculous rate and go out of business. That's an easy one.
  3. PR grows faster than the quality of your code. Result: everybody checks out your code, and it's not good yet. These people will be permanently convinced that your code is simple and inadequate, even if you improve it drastically later. I call this the Marimba phenomenon . Or, you get PR before there's a product people can buy, then when the product really comes out the news outlets don't want to do the story again. We'll call this the Segway phenomenon.
  4. Employees grows faster than code: Result: too many cooks working on code in the early days causes bad architecture. Software development works best when a single person creates the overall architecture and only later parcels out modules to different developers. And if you add developers too fast, development screeches to a halt, a phenomenon well understood since 1975 .

And so on, and so on... A small company growing at a natural pace has a reasonable chance of keeping these things in balance. But VCs don't like the flat part of the curve at the beginning, because they need an exit strategy in which the hockey-stick part of the curve occurs before their fund needs to cash out, about six years according to VC Joi Ito . This is in direct conflict with the fact that good software can't really accomplish this kind of growth. Hockey stick, there will be, but it will take longer than most VCs are willing to wait. Remember my chart of Lotus Notes? Good heavens, I am repeating myself.

VCs try to speed things up by spending more money. They spend it on PR, and then you get problem 3 ("PR grows faster than code"). They spend it on employees, and then you get problem 4 ("too many cooks") and problem 2 ("high burn rate"). They hire HR people, marketing people, business development people. They spend money on advertising. And the problem is, they spend all this money before anyone has had a chance to learn what the best way to spend money is. So the business development guy wanders around aimlessly and accomplishes zilch. You advertise in magazines that VCs read, not magazines that your customers read. And so on.

OK, that's the first part of the VC crisis.

The second part is the fact that VCs hear too many business plans, and they need to reject 999 out of 1000. There appear to be an infinite number of business plans looking for funding. A VC's biggest problem is filtering the incoming heap to find what they consider to be that needle in the haystack that's worth funding. So they get pretty good at saying "no," but they're not so good at saying no to the bad plans and yes to the good plans.

When you have to say ?euro;œno?euro;? 999 times for every time you say ?euro;œyes,?euro;? your method becomes whack-a-mole. Find the flaw, say no. Find the flaw, say no. The faster you find flaws, the more business plans you can ding. Over at VentureBlog you can amuse yourself for an hour with some of the trivial reasons VCs will ding you. PowerPoint too complicated? Ding! Won't tell us your magic sauce? Ding! You didn't research the VC before you came in? Ding! It's not their fault; they are just trying to say no 999 times in as efficient a way as possible. All of this reminds me too much of the old-school manager who hires programmers based on what school they went to or whether they look good in a suit.

Naval Ravikant, a VC at August Capital, reveals the classic VC myopia of feeling like they just don't have time to get to know entrepreneurs that aren't ready to pitch yet. ?euro;œMost VCs are too busy to 'dance,'?euro;? he wrote. They are too busy vetting serious proposals to shmooze with interesting companies that might not need cash right now.

This is, roughly, the equivalent of the old joke about the guy searching for his car keys under a streetlamp. "Did you lose them here?" asks the cop. "No, I lost them over there, but the light's better here."

But the great companies are often not the ones that spend all their time begging for investments. They may already be profitable. They may be too busy to look for VC, something which is a full time job for many entrepreneurs. Many excellent entrepreneurs feel that their time is better spent pitching products to customers rather than pitching stock to investors. It's bizarre that so many VCs are willing to ignore these companies simply because they aren't playing the traditional get-funded game. Get out there and pursue them!

Here's another funny thing that's happening. VCs are reacting to the crash by demanding ever stricter conditions for investments. It's now considered standard that the VC gets all their money back before anyone else sees a dime, no matter what percent of the company they actually own. VCs feel like this protects their interests. What they're forgetting is that it reduces the quality of startups that are willing to make deals. Here's one of VC Joi Ito's suggestion for VCs : ?euro;œSign a 'no shop' and get a letter of intent (LOI) signed quickly so an auction doesn't start jacking up the price.?euro;? A no shop is sometimes called an exploding term sheet. It means that the company must either accept the deal on the spot or it won't get funded at all. The theory is, we don't want you going around to other VCs trying to get a better deal. It's common among the second-tier VCs, but the best VCs are usually willing to stand on their own merits.

It seems to me that a company that accepts an exploding offer is demonstrating a remarkable lack of basic business aptitude. Every building contractor in New York knows you request bids from five or ten plumbers before you award the contract. If a plumber said, "I'll do it for $x, but if you shop around, deal's off," the contractor would laugh his head off and throw the plumber out on the street. Nothing sends a stronger message that an offer is uncompetitive than refusing to expose it to competition. And that's for a $6000 kitchen installation. Getting $10 million in funding for a business is the biggest and most important deal in the life of a company. You're going to be stuck with this VC forever, they're going to want to control your board of directors, they're going to push the founders out and bring in some polished CEO as quickly as they can, someone who will take the picture of the cat off your homepage and replace it with the usual MBA jargon.

And now they want you to agree to all this in a matter of fifteen minutes without talking to anyone else? Yeah, right.

VCs who make exploding offers are pretty much automatically eliminating all the people with good business sense from their potential universe of companies. Again, it does make it easier to say no 999 times, but you're pretty much guaranteed to say no to all the companies with a modicum of negotiating skills. This is not the correlation you're looking for. In fact, just about everything the VCs do to make their deals "tougher," like demanding more control, more shares, more preferential shares, lower valuations, death spiral convertible stock, etc., is pretty much guaranteed to be at the expense of the founders in a very zero-sum kind of way. And this means that smart founders, especially the ones with businesses that can survive a lack of funding, are going to walk away. VCs must realize that if the business flops, no matter how much control you have, the investor is going to lose everything. Look at the story of arsDigita. A nasty fight over control gives Phil Greenspun enough money to buy an airplane, and the VCs still lost every penny when the company went down the tubes. So all these tough deals are not really protecting the VCs, they're just restricting the VCs' world of possible investments to dumb companies and desperate companies. Sam Bhaumik, VC, says ?euro;œVCs are being aggressive, but most requests are legitimate.?euro;? The capital belongs to public pension funds and university endowments, he notes, using the standard widows-and-orphans sob story. Boo hoo . Come on , public pension funds and university endowments are the savviest investors out there; don't tell me they need coddling and protecting. They're investing in risky venture funds for a reason: they want to get paid for taking risk. If they wanted protection, they'd invest in US Treasuries.

There are probably hundreds of software companies started every day. Of that universe, there is a small number that are actively looking for early stage investors. Of that small number, an even smaller portion is willing to go along with the current harsh deals that VCs are offering. Now slice away the founders who are afraid of being arsDigita'd. The population shrinks even more as VCs reject companies that don't match their?euro;”quite reasonable?euro;”criteria for spotting a successful company. You wind up with a tiny number of investment opportunities which, quite frankly, is vanishingly unlikely to contain The Next Netscape.

More Reading

Considering VC? First read this article on the web:

An Engineer's View of Venture Capitalists , by Nick Tredennick

Don't miss these three books by company founders:

  • High St@kes, No Prisoners: A Winner's Tale of Greed and Glory in the Internet Wars by Charles Ferguson.
  • The Leap: A Memoir of Love and Madness in the Internet Gold Rush by Tom Ashbrook
  • Burn Rate: How I Survived the Gold Rush Years on the Internet by Michael Wolff
  • Startup: A Silicon Valley Adventure by Jerry Kaplan

A movie about the process:

  • Startup.com

And don't forget:

  • Eboys: The First Inside Account of Venture Capitalists at Work by Randall E. Stross

Weblogs by VCs:

  • VentureBlog
  • Joi Ito


Fog Creek Compensation

Wednesday, August 30, 2000

At Fog Creek Software, the way we make sure that people are paid fairly and rewarded for excellent work is based on a professional ladder.

Historically, our ladder is based on Microsoft's professional ladder, which was adopted and publicized by Construx. Many of the guidelines you see here are blatantly copied from there, with minor modifications for the culture we are trying to create. The categories also correspond closely to standard US corporate compensation practices as documented in this book by Michael O'Malley.

Fog Creek explicitly recognizes that many good software engineers have no desire whatsoever to do "management" or to take on a formal personnel management role. One of the purposes of the Fog Creek Professional Ladder is to create a career path with promotions for engineers who simply do not want to do management stuff at all. We want to avoid the trap that many companies fall into of forcing good programmers to stop programming and start managing people, if that's not what they want to do.

For various irrational reasons, nobody likes to be a level 1 or 2, so HR departments have always historically started the numbering with a higher number. So all Fog Creek software engineers are ranked at one of several different levels, level 8 through level 15.

Briefly:

  • Level 8 is for interns.
  • Level 9 is where new college grads, or people new to the industry, generally start. Then they work their way up to:
  • Level 12, which is full professional status. Most people won't go beyond this level.
  • Level 13-15 can be achieved only by making significant, above average contributions both to Fog Creek and within the field of Computer Science.

Our official titles will be:

(8) Technical Intern
(9-12) Member of Technical Staff
(13-15) Fog Creek Fellow

"Member of Technical Staff", a.k.a. MTS, is the classic title that all researchers at Bell Labs were given in their heyday. It's neat and egalitarian.

When we decide to hire someone, the interviewers who were involved in the "hire" decision will sit down with the candidates resume and salary history and figure out what level they will start at. After that, every six months, Fog Creek management will review the performance of everyone in the company with the goal of making sure that people are at the right level. It is most important to make sure that people are grouped in the same level with other engineers who are their true peers. At this point it might be appropriate to promote some people to a higher level. We will also compare our salaries within each level to the competitive norms to make sure we are paying market salaries, at which point people might receive a raise.


Foreword to Painless Project Management with FogBugz, by Mike Gunderloy

Friday, February 11, 2005

There?euro;™s a restaurant in my New York City neighborhood called Isabella?euro;™s that?euro;™s always packed.

Downstairs, upstairs, at the sidewalk cafe, it?euro;™s mobbed. And there are large crowds of happy yuppies out front, waiting 45 minutes for a table when they can clearly see other perfectly good restaurants right across the street which have plenty of tables.

It doesn?euro;™t matter when you go there. For Sunday brunch, it?euro;™s packed. Friday night? Packed, of course. But go on a quiet Wednesday night at 11:00 PM. You?euro;™ll get a table fairly quickly, but the restaurant is still, basically, packed.

Is it the food? Nah. Ruth Reichl, restaurant reviewer extraordinaire from the New York Times, dismissed it thusly: ?euro;œThe food is not very good.?euro;?

The prices? I doubt anyone cares. This is the neighborhood where Jerry Seinfeld bought Isaac Stern?euro;™s apartment with views over two parks.

Lack of competition? What, are you serious? This is Manhattan!

Here?euro;™s a clue as to why Isabella?euro;™s works. In ten years living in this neighborhood, I still go back there. All the time. Because they?euro;™ve never given me a single reason not to.

That actually says a lot.

I never go to a certain fake-Italian art-themed restaurant, because once I ate there and the waiter, who had gone beyond rude well into the realm of actual cruelty, mocking our entree choices, literally chased us down the street complaining about the small tip we left him.

I stopped going to another hole-in-the-wall pizza-pasta-bistro because the owner would come sit down at our table while we ate and ask for computer help.

I really, really loved the food at a local curry restaurant with headache-inducing red banquettes and zebra-striped decor. The katori chat was to die for. I was even willing to overlook the noxious smell of ammonia wafting up from the subterranean bathrooms. But the food inevitably took an hour to arrive, even when the place was empty, so I just never went back.

But in ten years I can?euro;™t think of a single bad thing that ever happened to me at Isabella?euro;™s.

Nothing.

So that?euro;™s why it?euro;™s so packed. People keep coming back, again and again, because when you dine at Isabella?euro;™s, nothing will ever go wrong.

Isabella?euro;™s is thoroughly and completely debugged.

It takes you ten years to notice this, because most of the time when you eat at a restaurant, nothing goes wrong. It took a couple of years of going to the curry place before we realized they were always going to make us miss our movie, no matter how early we arrived, and we finally had to write them off.

And so, on the Upper West Side of Manhattan, if you?euro;™re a restaurant, and you want to thrive, you have to carefully debug everything.

You have to make sure that there?euro;™s always someone waiting to greet guests. This person must learn never to leave the maitre d?euro;™ desk to show someone to their table, because otherwise the next person will come in and there will be nobody there to greet them. Instead, someone else needs to show patrons to their tables. And give them their menus, right away. And take their coats and drink orders.

You have to figure out who your best customers are?euro;”the locals who come on weekday nights when the restaurant is relatively quiet?euro;”and give them tables quickly on Friday night, even if the out-of-towners have to wait a little longer.

You need a procedure so that every water glass is always full.

Every time somebody is unhappy, that?euro;™s a bug. Write it down. Figure out what you?euro;™re going to do about it. Add it to the training manual. Never make the same mistake twice.

Eventually, Isabella?euro;™s became a fabulously profitable and successful restaurant, not because of its food, but because it was debugged. Just getting what we programmers call ?euro;œthe edge cases?euro;? right was sufficient to keep people coming back, and telling their friends, and that?euro;™s enough to overcome a review where the New York Times calls your food ?euro;œnot very good.?euro;?

Great products are great because they?euro;™re deeply debugged. Restaurants, software, it?euro;™s all the same.

Great software doesn?euro;™t crash when you do weird, rare things, because everybody does something weird.

Microsoft developer Larry Osterman, working on DOS 4, once thought he had found a rare bug. ?euro;œBut if that were the case,?euro;? he told DOS architect Gordon Letwin, ?euro;œit?euro;™d take a one in a million chance for it to happen.?euro;?

Letwin?euro;™s reply? ?euro;œIn our business, one in a million is next Tuesday.?euro;?

Great software helps you out when you misunderstand it. If you try to drag a file to a button in the taskbar, Windows pops up a message that says, essentially, ?euro;œYou can?euro;™t do that!?euro;? but then it goes on to tell you how you can accomplish what you?euro;™re obviously trying to do (try it!)

Great software pops up messages that show that the designers have thought about the problem you?euro;™re working on, probably more than you have. In FogBugz, for example, if you try to reply to an email message but someone else tries to reply to that same email at the same time, you get a warning and your response is not sent until you can check out what?euro;™s going on.

Great software works the way everybody expects it to. I?euro;™m probably one of the few people left who still closes windows by double clicking in the top left corner instead of clicking on the [x] button. I don?euro;™t know why I do that, but it always works, with great software. Some software that I have is not so great. It doesn?euro;™t close if you double click in the top left corner. That makes me a little bit frustrated. It probably made a lot of people frustrated, and a lot of those people probably complained, but I?euro;™ll bet you that the software developers just didn?euro;™t do bug tracking, because they have never fixed that bug and probably never will.

What great software has in common is being deeply debugged and the only way to get software that?euro;™s deeply debugged is to keep track of your bugs.

A bug tracking database is not just a memory aid, or a scheduling tool. It doesn?euro;™t make it easier to produce great software, it makes it possible to create great software.

With bug tracking, every idea gets into the system. Every flaw gets into the system. Every tester?euro;™s possible misinterpretation of the user interface gets into the system. Every possible improvement that anybody thinks about gets into the system.

Bug tracking software captures the cosmic rays that cause the genetic mutations that make your software evolve into something superior.

And as you constantly evaluate, reprioritize, triage, punt, and assign these flaws, the software evolves. It gets better and better. It learns to deal with more and more weird situations, more and more misunderstanding users, and more and more scenarios.

That?euro;™s when something magical happens, and your software becomes better than just the sum of its features. Suddenly it becomes reliable. Reliable, meaning, it never screws up. It never makes its users angry. It never makes its customers wish they had purchased something else.

And that magic is the key to success. In restaurants as in software.


Free Desktop Pictures!

Friday, August 04, 2000

Joel on Software presents... free desktop pictures!

Stuck behind a desk? Brighten your monitor a bit with these high resolution digital pictures I took this summer in East Hampton.


Getting Things Done When You're Only a Grunt

Tuesday, December 25, 2001

This site is supposed to be about software management. But sometimes you don't have the power to create change in your organization by executive fiat. Obviously, if you're just a grunt programmer at the bottom of the totem pole, you can't exactly order people to start creating schedules or bug databases. And in fact even if you're a manager, you've probably discovered that managing developers is a lot like herding cats, only not as fun. Merely saying "make it so" doesn't make it so.

It can be frustrating when you're working in an organization that scores low on The Joel Test. No matter how good your code is, your coworkers write such bad code that you're embarassed to be associated with the project. Or management is making bad decisions about what code to write, so you're forced to waste your talent debugging the AS/400 version of a retirement-planning game for kids.

You could just leave, I suppose. But presumably, there's some reason you're stuck there. The stock options haven't quite vested, there's no better place to work in Podunk, or perhaps your boss is holding someone you love hostage. In any case, dealing with life on a bad team can be infuriating. But there are strategies for improving your team from the bottom, and I'd like to share a few of them.

Strategy 1 Just Do It

A lot can be done to improve the project just by one person doing it. Don't have a daily build server? Make one. Set your own machine up with a scheduled job to make builds at night and send out email results. Does it take too many steps to make the build? Write the makefile. Nobody does usability tests? Do your own hallway usability tests on the mailroom folks with a piece of paper or a VB prototype.

Strategy 2 Harness the Power of Viral Marketing

Many of the Joel Test strategies can be implemented by a single person on an uncooperative team. Some of them, if done well, will spread to the rest of the team.

For example, suppose nobody on your team can be persuaded to use a bug database. Don't let it bother you. Just keep your own. Enter bugs that you find in your own code. If you find a bug that somebody else really should fix, assign the bug to them using the bug database. If you have good bug tracking software, this will send them an email. But now, you can keep sending them emails if they don't fix the bug. Eventually, they'll see the value of bug tracking and start to use the system as it was intended. If the QA team refuses to input bugs to the bug tracking system, simply refuse to listen to bug reports through any other channel. About the three-thousdandth time that you say to people, "listen, I'd love to fix that, but I'm going to forget. Can you enter a bug in the system?" they'll start using the database.

Nobody on your team wants to use source control? Create your own CVS repository, on your own hard drive if necessary. Even without cooperation, you can check your code in independently from everybody else's. Then when they have problems that source control can solve (someone accidentally types rm * ~ instead of rm *~), they'll come to you for help. Eventually, people will realize that they can have their own checkouts, too.

Strategy 3 Create a Pocket of Excellence

The team won't make schedules? Or specs? Write your own. Nobody's going to complain if you take a day or two to write a minimal spec and schedule for the work you're about to do.

Get better people into the team. Get involved in hiring and interviewing, and recruit good candidates to join the team.

Find the people who are willing to improve and capable of it, and get them on your side. Even on poor teams, you're likely to have some smart people who just don't have the experience to create great code. Help them out. Set them up to learn. Read their code checkins. If they do something stupid, don't send them a snooty email explaining what's stupid about their checkins. That will just make them angry and defensive. Instead, innocently report the bug that you know is the result of the checkin. Let them figure out what's causing it. When they find the bug for themselves, they'll remember that lesson a lot better.

Strategy 4 Neutralize The Bozos

Even the best teams can have a bozo or two. The frustrating part about having bad programmers on your team is when their bad code breaks your good code, or good programmers have to spend time cleaning up after the bad programmers.

As a grunt, your goal is damage-minimization, a.k.a. containment. At some point, one of these geniuses will spend two weeks writing a bit of code that is so unbelievably bad that it can never work. You're tempted to spend the fifteen minutes that it takes to rewrite the thing correctly from scratch. Resist the temptation. You've got a perfect opportunity to neutralize this moron for several months. Just keep reporting bugs against their code. They will have no choice but to keep slogging away at it for months until you can't find any more bugs. Those are months in which they can't do any damage anywhere else.

Strategy 5 Get Away From Interruptions

All happy work environments are alike (private offices, quiet working conditions, excellent tools, few interruptions and even fewer large meetings). All unhappy work environments are unhappy in their own way.

The bad news is that changing the working environment is almost impossible in virtually any company. Long term leases may mean that even the CEO can't do anything about it. That's why so few software developers get private offices. This hurts their companies in at least two ways. First, it makes it harder to recruit top notch developers, who will prefer the firm that gives them cushier conditions (all else being equal). Second, the level of interruptions can dramatically reduce the productivity of developers, who find it impossible to get into the zone and stay in it for any length of time.

Look for ways to get out of this environment. Take a laptop to the company cafeteria, where there are lots of tables that are empty most of the day (and nobody can find you). Book a conference room for the whole day and write code there, and make it clear through the preponderance of checkins just how much more work you get done when you're in a room by yourself. The next time there's a crunch on and your manager asks you what you need to Get This Done By Tomorrow, you know what to say. They'll find you an office for the day. And pretty soon they'll start wondering what they can do to keep that productive thing going year round.

Come into work late and leave late. Those hours after the rest of the company goes home can be the most productive. Or, if you're on a team of developers who regularly come in late, get into work at 9 am. You'll do more work in the two hours before other people come in and start bothering you than you do in the rest of the day.

Don't keep your email or IM client running. Check your email every hour, if you want, but don't keep it running.

Strategy 6 Become Invaluable

None of these strategies work if you're not really an excellent contributor. If you don't write good code, and lots of it, you're just going to be resented for messing around with bug databases when you "should be" writing code. There's nothing more deadly to your career than having a reputation of being so concerned with process that you don't accomplish anything.

Once, when I started a new job as a grunt programmer at a new company, I discovered that the company was running somewhere around 2 on the Joel Test, and I was determined to fix it. But I also knew that making a good first impression was crucial. So I allocated the first seven hours of every day to just writing code, as was expected of me. There's nothing like a flurry of checkins to make you look good to the rest of the development team. But I reserved another hour every afternoon before going home to improving the process. I used that time to fix things that made it hard to debug our product. I set up a daily build and a bug database. I fixed all the longstanding annoyances that made development difficult. I wrote specs for the work that I was doing during the day. I wrote a document explaining step-by-step how to create a development machine from scratch. I thoroughly documented an important internal language which had been undocumented. Slowly, the process got better and better. Nobody but me (and my team, when I was put in charge of one) ever did schedules or specs, but other than that we hit about 10 on the Joel Test.

You can make things better, even when you're not in charge, but you have to be Caesar's Wife: above suspicion. Otherwise you'll make enemies as you go along.

Any other ideas?


Good Software Takes Ten Years. Get Used To it.

Saturday, July 21, 2001

Have a look at this little chart:

[Source: Iris Associates]

This is a chart showing the number of installed seats of the Lotus Notes workgroup software, from the time it was introduced in 1989 through 2000. In fact when Notes 1.0 finally shipped it had been under development for five years. Notice just how dang long it took before Notes was really good enough that people started buying it. Indeed, from the first line of code written in 1984 until the hockey-stick part of the curve where things really started to turn up, about 11 years passed. During this time Ray Ozzie and his crew weren't drinking piña coladas in St Barts. They were writing code.

The reason I'm telling you this story is that it's not unusual for a serious software application. The Oracle RDBMS has been around for 22 years now. Windows NT development started 12 years ago. Microsoft Word is positively long in the tooth; I remember seeing Word 1.0 for DOS in high school (that dates me, doesn't it? It was 1983.)

To experienced software people, none of this is very surprising. You write the first version of your product, a few people use it, they might like it, but there are too many obvious missing features, performance problems, whatever, so a year later, you've got version 2.0. Everybody argues about which features are going to go into 2.0, 3.0, 4.0, because there are so many important things to do. I remember from the Excel days how many things we had that we just had to do. Pivot Tables. 3-D spreadsheets. VBA. Data access. When you finally shipped a new version to the waiting public, people fell all over themselves to buy it. Remember Windows 3.1? And it positively, absolutely needed long file names, it needed memory protection, it needed plug and play, it needed a zillion important things that we can't imagine living without, but there was no time, so those features had to wait for Windows 95.

But that's just the first ten years. After that, nobody can think of a single feature that they really need. Is there anything you need that Excel 2000 or Windows 2000 doesn't already do? With all due respect to my friends on the Office team, I can't help but feel that there hasn't been a useful new feature in Office since about 1995. Many of the so-called "features" added since then, like the reviled ex-paperclip and auto-document-mangling, are just annoyances and O'Reilly is doing a nice business selling books telling you how to turn them off.

So, it takes a long time to write a good program, but when it's done, it's done. Oh sure, you can crank out a new version every year or two, trying to get the upgrade revenues, but eventually people will ask: "why fix what ain't broken?"

Failure to understand the ten-year rule leads to crucial business mistakes.

Mistake number 1. The Get Big Fast syndrome. This fallacy of the Internet bubble has already been thoroughly discredited elsewhere, so I won't flog it too much. But an important observation is that the bubble companies that were trying to create software (as opposed to pet food shops) just didn't have enough time for their software to get good. My favorite example is desktop.com, which had the beginnings of something that would have been great if they had worked on it for 10 years. But the build-to-flip mentality, the huge overstaffing and overspending of the company, and the need to raise VC every ten minutes made it impossible to develop the software over 10 years. And the 1.0 version, like everything, was really morbidly awful, and nobody could imagine using it. But desktop.com 8.0 might have been seriously cool. We'll never know.

Mistake number 2. the Overhype syndrome. When you release 1.0, you might want to actually keep it kind of quiet. Let the early adopters find it. If you market it and promote it too heavily, when people see what you've actually done, they will be underwhelmed. Desktop.com is an example of this, so is Marimba, and Groove: they had so much hype on day one that people stopped in and actually looked at their 1.0 release, trying to see what all the excitement was about, but like most 1.0 products, it was about as exciting as watching grass dry. So now there are a million people running around who haven't looked at Marimba since 1996, and who think it's still a dorky list box that downloads Java applets that was thrown together in about 4 months.

Keeping 1.0 quiet means you have to be able to break even with fewer sales. And that means you need lower costs, which means fewer employees, which, in the early days of software development, is actually a really great idea, because if you can only afford 1 programmer at the beginning, the architecture is likely to be reasonably consistent and intelligent, instead of a big mishmash with dozens of conflicting ideas from hundreds of programmers that needs to be rewritten from scratch (like Netscape, according to the defenders of the decision to throw away all the source code and start over).

Mistake number 3. Believing in Internet Time. Around 1996, the New York Times first noticed that new Netscape web browser releases were coming out every six months or so, much faster than the usual 2 year upgrade cycle people were used to from companies like Microsoft. This led to the myth that there was something called "Internet time" in which "business moved faster." Which would be nice, but it wasn't true. Software was not getting created any faster, it was just getting released more often. And in the early stages of a new software product, there are so many important things to add that you can do releases every six months and still add a bunch of great features that people Gotta Have. So you do it. But you're not writing software any faster than you did before. (I will give the Internet Explorer team credit. With IE versions 3.0 and 4.0 they probably created software about ten times faster than the industry norm. This had nothing to do with the Internet and everything to do with the fact that they had a fantastic, war-hardened team that benefited from 15 years of collective experience creating commercial software at Microsoft.)

Mistake number 4. Running out of upgrade revenues when your software is done. A bit of industry lore: in the early days (late 1980s), the PC industry was growing so fast that almost all software was sold to first time users. Microsoft generally charged about $30 for an upgrade to their $500 software packages until somebody noticed that the growth from new users was running out, and too many copies were being bought as upgrades to justify the low price. Which got us to where we are today, with upgrades generally costing 50%-60% of the price of the full version and making up the majority of the sales. Now the trouble comes when you can't think of any new features, so you put in the paperclip, and then you take out the paperclip, and you try to charge people both times, and they aren't falling for it. That's when you start to wish that you had charged people for one year licenses, so you can make your product a subscription and have permission to keep taking their money even when you haven't added any new features. It's a neat accounting trick: if you sell a software package for $100, Wall Street will value that at $100. But if you can sell a one year license for $30, then you can claim that you're going to get recurring revenue of $30 for the next, say, 10 years, which is worth $200 to Wall Street. Tada! Stock price doubles! (Incidentally, that's how SAS charges for their software. They get something like 97% renewals every year.)

The trouble is that with packaged software like Microsoft's, customers won't fall for it. Microsoft has been trying to get their customers to accept subscription-based software since the early 90's, and they get massive pushback from their customers every single time. Once people got used to the idea that you "own" the software that you bought, and you don't have to upgrade if you don't want the new features, that can be a big problem for the software company which is trying to sell a product that is already feature complete.

Mistake number 5. The "We'll Ship It When It's Ready" syndrome. Which reminds me. What the hell is going on with Mozilla? I made fun of them more than a year ago because three years had passed and the damn thing was still not out the door. There's a frequently-obsolete chart on their web site which purports to show that they now think they will ship in Q4 2001. Since they don't actually have anything like a schedule based on estimates, I'm not sure why they think this. Ah, such is the state of software development in Internet Time Land.

But I'm getting off topic. Yes, software takes 10 years to write, and no, there is no possible way a business can survive if you don't ship anything for 10 years. By the time you discount that revenue stream from 10 years in the future to today, you get bupkis, especially since business analysts like to pretend that everything past 5 years is just "residual value" when they make their fabricated, fictitious spreadsheets that convince them that investing in sock puppets at a $100,000,000 valuation is a pretty good idea.

Anyway, getting good software over the course of 10 years assumes that for at least 8 of those years, you're getting good feedback from your customers, and good innovations from your competitors that you can copy, and good ideas from all the people that come to work for you because they believe that your version 1.0 is promising. You have to release early, incomplete versions -- but don't overhype them or advertise them on the Super Bowl, because they're just not that good, no matter how smart you are.

Mistake number 6. Too-frequent upgrades (a.k.a. the Corel Syndrome). At the beginning, when you're adding new features and you don't have a lot of existing customers, you'll be able to release a new version every 6 months or so, and people will love you for the new features. After four or five releases like that, you have to slow down, or your existing customers will stop upgrading. They'll skip releases because they don't want the pain or expense of upgrading. Once they skip a release, they'll start to convince themselves that, hey, they don't always need the latest and greatest. I used Corel PhotoPaint 6.0 for 5 years. Yes, I know, it had all kinds of off-by-one bugs, but I knew all the off-by-one bugs and compensated by always dragging the selection one pixel to the right of where I thought it should be.

Make a ten year plan. Make sure you can survive for 10 years, because the software products that bring in a billion dollars a year all took that long. Don't get too hung up on your version 1 and don't think, for a minute, that you have any hope of reaching large markets with your first version. Good software, like wine, takes time.


Hard-assed Bug Fixin'

Tuesday, July 31, 2001

Software quality, or the lack thereof, is something everybody loves to gripe about. Now that I have my own company I finally decided to do something about it. Over the last two weeks we stopped everything at Fog Creek to ship a new incremental version of FogBUGZ with the goal of eliminating all known bugs (there were about 30).

As a software developer, fixing bugs is a good thing. Right? Isn't it always a good thing?

No!

Fixing bugs is only important when the value of having the bug fixed exceeds the cost of the fixing it.

These things are hard, but not impossible, to measure. Let me give you an example. Suppose you operate a peanut-butter-and-jelly sandwich factory. Your factory produces 100,000 sandwiches a day. Recently, due to the introduction of some new flavors (garlic peanut butter with spicy Habanero jam), demand for your product has gone through the roof. The factory is operating full-out at 100,000 sandwiches, but the demand is probably closer to 200,000. You just can't make any more. And each sandwich earns you a profit of 15 cents. So you're losing $15,000 a day in potential earnings because you don't have enough capacity.

Building a new factory would cost way too much. You don't have the capital, and you're afraid that spicy/garlicky sandwiches are just a fad which will pass, anyway. But you're still losing that $15,000 a day.

It's a good thing you hired Jason. Jason is a fourteen year old programmer who hacked into the computers that run the factory, and believes that he has come up with a way to speed up the assembly line by a factor of 2. Something about overclocking that he heard on slashdot. And it seemed to work in a test run.

There's only one thing stopping you from rolling it out. There's a teeny tiny wee little bug that causes a sandwich to be mushed once an hour or so. Jason wants to fix the wee bug. He thinks he can fix it in three days. Do you let him fix it, or do you roll out the software in its bug-addled state?

Rolling out the software three days later will cost you $45,000 in lost profits. And it will save you, um, the cost of raw materials for 72 sandwiches. (In either case Jason will get the bug fixed three days later). Well, I don't know how much sandwiches cost on your planet, but here on Earth, they're a lot less than $625.

Where was I. Oh yeah. Sometimes it is not worth fixing a bug. Here's another bug that's not worth fixing: if you have a bug that totally crashes your program when you open gigantic files, but it only happens to your single user who has OS/2 and who, for all you know, doesn't even use large files. Well, don't fix it. Worse things have happened at sea. Similarly I've generally given up caring about people with 16 color screens or people running off-the-shelf Windows 95 with no upgrades in 7 years. People like that don't spend much money on packaged software products. Trust me.

But mostly, it's worth fixing bugs. Even if they are "harmless" bugs, they may reduce the reputation of your company and your product, which, in the long run, will have a significant impact on your earnings. It's hard to overcome the reputation of having a buggy product. When you do want to do that .01 release, here are some ideas for finding and fixing the right bugs: the ones that it is economically worth fixing.

Step One: Make Sure You Find Out About The Bugs.

In the case of FogBUGZ, we have two ways of doing that. First, we trap all bugs on our free demo server, capture as much information as we can, and email the whole thing to the development team. That found an awful lot of bugs, which was very cool. For example, we discovered a bunch of people who didn't enter dates where they were supposed to in the "Fix For" screen. We didn't even have an error message in that case, we just "crashed" (which, in a web app, just means you got an ugly IIS error instead of what you expected). Oops.

When I worked at Juno, we had an even cooler system in place to collect bugs "from the field" automatically. We installed a handler using TOOLHELP.DLL so that every time Juno crashed, it stayed alive just long enough to dump the stack into a log file before going to its grave. The next time the program connected to the Internet to send mail, it uploaded the log file. During betas, we gathered these log files, collated all the crashes, and entered them into the bug tracking database. This found literally hundreds of crashing bugs. When you have a million users, it is amazing what will crash, often because of severe low memory conditions or severely crappy computers (can you spell Packard Bell?) You could have code like this:

int foo( object& r )
{
r.Blah();
return 1;
}

and you would get crashes there because the r reference was NULL, even though that's completely impossible, there's no such thing as a NULL reference, C++ guarantees it, and you don't have to believe me but when you wait long enough and have millions of users and religiously collect their stack dumps, you will find crashes in places like that and you won't believe your eyes. (And you won't fix them. Cosmic rays, man. Get a new computer and this time don't install every cool shareware taskbar lint gizmo you find. Sheesh.)

The other thing we do is consider each and every tech support call to be evidence of a bug. When we take the call, we try to figure out what we could have done to eliminate it. For example, the old FogBUGZ Setup used to assume that FogBUGZ would run under the anonymous Internet user account. That was a good assumption 95% of the time, and a bad assumption 5% of the time, but every one of those 5% cases ended up in a call to our support line. So we modified Setup to prompt for an account.

Step Two: Make Sure You Get Economic Feedback

You may not be able to figure out exactly how much it's worth to fix each bug, but there's something you can do: charge the "cost" of tech support back to the business unit. In the early nineties there was a financial reorganization at Microsoft under which each product unit was charged for the full cost of all tech support calls. So the product units started insisting that PSS (Microsoft's tech support) provide lists of Top Ten Bugs regularly. When the development team concentrated on those, product support costs plummeted.

This is a bit in contradiction with the new trend of letting the tech support department pay for its own operation, something that most large companies do. At Juno tech support was expected to break even by charging people for tech support. By moving the economic burden of bugs onto the users themselves, you lose what limited ability you might have had to detect the damage they were causing. (Instead you get irate users who resent having to pay for your bug, who tell their friends, and you can't even measure how much that costs you. To be fair to Juno, the product itself was free, so stop yer bitchin.)

One way of resolving the two is to not charge the user when the support call was caused by a bug in your own product. Microsoft does this, and it's quite nice, and I've never paid for a call to Microsoft :) Instead, charge the $245 or whatever one developer incident costs these days back to the product unit. That blows away their profit completely for the product they sold you (several times over), and creates exactly the right economic incentives. Which reminds me of one reason DOS games were a terrible business... to get them to look good and run fast, you usually needed strange video drivers, and a single tech support call about the video drivers would blow away the profit you could make from 20 copies of your product, assuming Egghead and Ingram and the ad on MTV hadn't already guzzled away all your earnings.

Step Three: Figure Out What It's Worth To You To Fix Them All.

At Fog Creek Software, well, we're a tiny company (except in our own minds), and the development team just takes the tech support calls. The cost was running about 1 hour per day, which, based on our consulting rates, is somewhere around $75000 a year. We were pretty confident that we could get that down to 15 minutes a day by fixing all known bugs.

Using very sloppy numbers, here, that means that the net present value of the savings would be about $150,000. That justifies 62 days of work: if you can do it in less than 62 person-days, it's worth doing.

Using the handy estimation feature built into FogBUGZ, we calculated that it would take 20 person-days (two people two weeks) to fix everything - that's $48,000 "spent" for a return of $150,000, which is a great return on investment just on the basis of the tech support savings. (Observe that you could substitute the cost of programmer's salaries and overhead instead of our consulting rate and get the same 3:1 result, since it cancels out).

I haven't even begun to count the value from having a better product, but I can start doing that, too. We had 55 crashes on the demo server during the month of July with the old code, representing 17 distinct users. You have to imagine that at least one of those people decided not to buy FogBUGZ because they thought it was buggy when they ran the demo (although I don't have real statistics for that.) In any case the lost sales was probably costing us somewhere between $7,000 and $100,000 in present value. (If you were serious enough, it wouldn't be too hard to get a real number).

Next question. Can you charge more for a less buggy product? That would add a whole bunch of value to debugging. I suspect that at the extremes, bug count does affect price, but I am hard pressed to think of an example from the world of packaged software where this has been the case.

Please Don't Beat Me Up!

Inevitably people read essays like this and come to silly conclusions, like, Joel doesn't think you should fix bugs. In fact I think that for most of the kinds of bugs that most people fix, there's a clear return on investment. But there may be an even higher monetary value to doing something other than fixing every last bug. If you have to decide between fixing the bug for OS/2 guy and adding a new feature that will sell 20,000 copies of your software to General Electric, well, sorry, OS/2 guy. And if you're dumb enough to think that it's still more important to fix OS/2 than to add the GE feature, maybe your competitors won't be and you'll be out of business.

With all that said, I'm optimistic at heart, and I believe that there is a lot of hidden value to producing very high quality products that is not very easy to capture. Your employees will be prouder. Fewer of your customers will send you back your CD in the mail after microwaving it and chopping it to bits with an ax. So I tend to err on the side of quality (indeed, we fixed every known bug in FogBUGZ, not just the big bang ones) and take pride in that, and feel confident, by the complete elimination of errors from the demo server, that we have a rock-solid product.


How do You Compensate Programmers?

Sunday, August 27, 2000

Editor's note: This article originally appeared August 28th, 2000. Based on a tremendous amount of great feedback received from my readers, I've completely rewritten the policy. This article appears as a matter of historical record only.

How do you compensate programmers?

I'm designing the compensation package for programmers at Fog Creek Software, and I could use your advice!

Here's the tentative plan. I'd love to hear your feedback and discussion on any and all aspects of it.

Fog Creek Software hires programmers at six different levels, Level 9 through Level 14, which roughly corresponds to their level of experience, seniority, and initiative. Programmers are assigned a level when they are hired, and may move up or down in level at any point in time.

Levels are set approximately as follows. These are merely guidelines; the real way they are set is by comparing groups of people at various levels and then adjusting based on a consensus of how good they are.

Level

Years of experience

Technology exposure

Initiative and architectural skills

9

0

Two programming classes and incredible aptitude

Can develop code with significant mentoring and review. Intended mainly for summer interns and high school kids.

10

Recent grad ?euro;“ 1 year

Coding for one OS like Unix, two or three programming languages, no API experience

Can execute on a plan or write code that someone else specified, requires some mentoring

11

1 years ?euro;“ 3 years

2 platforms (e.g. WinAPI and Unix); RDBMS; two or three other programming platforms (like ASP, JSP, PHP, perl, etc)

Can work independently on a feature and can suggest improvements and shortcuts

12

3 ?euro;“ 5 years

Enough different technologies to be able to make excellent choices of architecture

Can design and architect a feature independently; given an interesting problem to work on, can architect, lead, and build the whole thing; can mentor less experienced developers

13

5 ?euro;“ 10 years

All the above, plus, knows enough about other disciplines to be able to lead specialists in those areas (e.g. networking, wireless, hardware, datacenter, telecom, etc)

Can conceive, design, architect, and lead a team to implement an entire product or company. CTO-like skills.

14

10 years plus

Significant experience inventing and architecting new technologies which are widely used throughout the industry

A "Fog Creek Fellow": Invents whole new programming languages like C++; extensively known through books and articles.

This level is reserved for hiring people like Bjarne Stroustrup or Linus Torvalds, or promoting geniuses to an independent research position.

Compensation consists of:

  1. Salary

  2. Annual bonus

  3. Benefits

  4. Stock options

  5. Startup bonus

Salary is intended to be very egalitarian, fair, and transparent. Fog Creek will have a simple schedule of salaries (set competitively) based on level. For reasons of fairness, there will be no variations on these salaries from person to person. Everybody knows what the guidelines are for levels; everybody knows what level they are at; and everybody knows what the salary is for their level.

Annual bonus is also intended to be equitable. Each year, Fog Creek management will determine what total dollar amount should be awarded in bonuses. This amount will be prorated according to salaries and percentage of the year that they worked, so every employee will receive a bonus amounting to an equal percentage of their salary: if I get a 11% bonus, everybody gets an 11% bonus. It is our explicit philosophy that bonuses are rewarded based on the performance of the entire company, not the individual. (Also, there is no penalty for quitting the company before bonuses are paid: if you do this, you?euro;™ll still get your prorated bonus for the part of the year that you worked. This policy is to prevent getting a wave of people quitting every January and screwing up our lives).

Benefits are equal for all full-time employees. (We may end up with categories like intern and temp with more limited benefits). Some employees may get more "value" out of their benefits than others (e.g., if they use the health clubs).

Stock Options are weighted heavily to compensate the people who take the most risk, namely, the people who join Fog Creek when it's just a wee tiny company. As the company grows, the initial package of stock options offered to new hires will be lower and lower. At any given time, stock option awards are equal based on level.

Since the basic compensation is set in stone, for the purpose of fairness and equity, the only tool we have to lure people who have great offers elsewhere is with a Startup bonus. This can vary wildly from new hire to new hire. It is mainly intended to compensate for cases where the normal benefits package does not appear to be competitive to a candidate who, for example, has an offer from another firm that is disproportional to their actual "market" value. As a typical example, suppose we're currently paying level 10s (new college hires) $60,000 a year + $5,000 startup bonus, and we think that's about fair. If we are having trouble luring a particular hire that we really want, because he or she has an offer from another company for $70,000, we'll just add $10,000 to the startup bonus and keep the salary at $60,000.

To summarize: as a company, we are very concerned with equity and fairness in salaries. We think it will be extremely valuable to maintain salaries, bonuses, and benefits for programmers at equal levels rather than negotiating individually with everyone and then having people upset that they are not getting a "fair" salary. So, at the point of trying to hire someone, if they simply wouldn't be happy with what the rest of the team is getting, we'll dazzle them with a startup bonus but set their actual salary where everyone else's is set. And if we're having trouble hiring because of low salaries, we'll raise the salaries for everyone, even the people we've already hired.

So, what do you think? Am I crazy? Is this naïve? Should I hire "compensation consultants" to charge me more than I bill annually just to set salaries? Let me know!

Read some responses


How Many Lies Can You Find In One Direct Mail Piece?

Saturday, March 24, 2001

George Meyer, one of the writers on the Simpsons, likes to collect examples of advertising in which "the word-to-falsehood ratio approaches one." In an interview in the New Yorker, he complained about a magazine ad for a butter substitute called Country Crock. "It's not from the country, there is no crock," he told the interviewer. "Two words, two lies."

Recently I got this piece of junk mail from Earthlink (larger view)

Just for my own personal amusement, I decided to try to count how many lies are on this particular fine specimen.

  1. The whole piece is attempting to disguise itself as an express mail letter. They are working on the assumption that ignorant recipients will confuse it with a Federal Express letter, or perhaps think that "Urgent Express" is a company that competes with FedEx. Uh huh. Ooooh! The marketing slimeballs at Earthlink must have drooled at the thought of old Marge in a nursing home, practically falling over with excitement that she got an urgent express letter, like those letters you get from fancy New York lawyers urging you to sue the nursing home.

  2. It says that it's urgent, and it isn't, it's just a stupid CD trying to get you to sign up for Internet access.

  3. It says that it's "express", but it's mailed using "Presorted Standard" rate, which isn't express at all, in fact, it probably cost precisely $0.23 to mail (FedEx is $15.34).

  4. It has a tracking number, but it is using a class of mail service that is not tracked, and I'll bet you anything that everybody gets the same tracking number. Sorry, folks, for $0.23 you don't get tracking.

  5. It uses a fake handwriting font, intended to make you think that it is not junk mail but rather something that a person made for you.

  6. It says that it's from Sky Dayton (founder of Earthlink) although it is actually from Earthlink's direct marketing department. I can just see the thought going through the copy writer's head. "People won't open it unless it's from someone they know, and everyone knows Sky Dayton!"

  7. It has a "sender's account number" which is, of course, completely fake (Anybody care to try to use it to sign up for Earthlink?)

  8. It has a little "payment" section with a checkbox indicating that it should be billed to the sender, even though this has no meaning on standard mail.

  9. It has a "release signature" with a little annotation that this signature is "required," when actually no "release signature" is required at all.

  10. It says "Time Sensitive Material Enclosed," which is almost certainly untrue and meaningless anyway.

  11. On the back, it says "Weight Limit 8 Ounces" even though the actual weight limit for Presorted Standard is 16 Ounces.

  12. It actually has a piece of clear plastic glued on the front. The clear plastic serves no purpose whatsoever except to imitate FedEx envelopes which have the same plastic to hold the mailing manifest, much like those little butterflies which have fierce markings in attempt to convince butterfly-eaters that they themselves are fierce, even though they couldn't be less fierce if they were dressed up like Julie Andrews in high heels singing "My Favorite Things." In nature this is called Batesian mimicry.

The biggest problem with direct mail (what we normal people call "Junk Mail") is that 99% of it (literally) gets thrown away unopened, Batesian mimicry or not. To combat this, direct mailers will do anything to get you to open their junk, no matter how dishonest.

When I went to Earthlink's home page, it only took me a couple of clicks to find their Mission and Core Values and Beliefs. And what it says there is:

We require complete honesty and integrity in everything we do.

I didn't even have the energy to open the damn thing and see how many lies were on the inside. Does this mailing strike you as "completely honest"? Or just "business as usual?" Am I too sensitive? Should I go back to complaining about complaining about Bloatware?

[Followup: Earthlink is unrepentant.]


How Microsoft Lost the API War

Sunday, June 13, 2004

Here's a theory you hear a lot these days: "Microsoft is finished. As soon as Linux makes some inroads on the desktop and web applications replace desktop applications, the mighty empire will topple."

Although there is some truth to the fact that Linux is a huge threat to Microsoft, predictions of the Redmond company's demise are, to say the least, premature. Microsoft has an incredible amount of cash money in the bank and is still incredibly profitable. It has a long way to fall. It could do everything wrong for a decade before it started to be in remote danger, and you never know... they could reinvent themselves as a shaved-ice company at the last minute. So don't be so quick to write them off. In the early 90s everyone thought IBM was completely over: mainframes were history! Back then, Robert X. Cringely predicted that the era of the mainframe would end on January 1, 2000 when all the applications written in COBOL would seize up, and rather than fix those applications, for which, allegedly, the source code had long since been lost, everybody would rewrite those applications for client-server platforms.

Well, guess what. Mainframes are still with us, nothing happened on January 1, 2000, and IBM reinvented itself as a big ol' technology consulting company that also happens to make cheap plastic telephones. So extrapolating from a few data points to the theory that Microsoft is finished is really quite a severe exaggeration.

However, there is a less understood phenomenon which is going largely unnoticed: Microsoft's crown strategic jewel, the Windows API, is lost. The cornerstone of Microsoft's monopoly power and incredibly profitable Windows and Office franchises, which account for virtually all of Microsoft's income and covers up a huge array of unprofitable or marginally profitable product lines, the Windows API is no longer of much interest to developers. The goose that lays the golden eggs is not quite dead, but it does have a terminal disease, one that nobody noticed yet.

Now that I've said that, allow me to apologize for the grandiloquence and pomposity of that preceding paragraph. I think I'm starting to sound like those editorial writers in the trade rags who go on and on about Microsoft's strategic asset, the Windows API. It's going to take me a few pages, here, to explain what I'm really talking about and justify my arguments. Please don't jump to any conclusions until I explain what I'm talking about. This will be a long article. I need to explain what the Windows API is; I need to demonstrate why it's the most important strategic asset to Microsoft; I need to explain how it was lost and what the implications of that are in the long term. And because I'm talking about big trends, I need to exaggerate and generalize.

Developers, Developers, Developers, Developers

Remember the definition of an operating system? It's the thing that manages a computer's resources so that application programs can run. People don't really care much about operating systems; they care about those application programs that the operating system makes possible. Word Processors. Instant Messaging. Email. Accounts Payable. Web sites with pictures of Paris Hilton. By itself, an operating system is not that useful. People buy operating systems because of the useful applications that run on it. And therefore the most useful operating system is the one that has the most useful applications.

The logical conclusion of this is that if you're trying to sell operating systems, the most important thing to do is make software developers want to develop software for your operating system. That's why Steve Ballmer was jumping around the stage shouting "Developers, developers, developers, developers." It's so important for Microsoft that the only reason they don't outright give away development tools for Windows is because they don't want to inadvertently cut off the oxygen to competitive development tools vendors (well, those that are left) because having a variety of development tools available for their platform makes it that much more attractive to developers. But they really want to give away the development tools. Through their Empower ISV program you can get five complete sets of MSDN Universal (otherwise known as "basically every Microsoft product except Flight Simulator") for about $375. Command line compilers for the .NET languages are included with the free .NET runtime... also free. The C++ compiler is now free. Anything to encourage developers to build for the .NET platform, and holding just short of wiping out companies like Borland.

Why Apple and Sun Can't Sell Computers

Well, of course, that's a little bit silly: of course Apple and Sun can sell computers, but not to the two most lucrative markets for computers, namely, the corporate desktop and the home computer. Apple is still down there in the very low single digits of market share and the only people with Suns on their desktops are at Sun. (Please understand that I'm talking about large trends here, and therefore when I say things like "nobody" I really mean "fewer than 10,000,000 people," and so on and so forth.)

Why? Because Apple and Sun computers don't run Windows programs, or, if they do, it's in some kind of expensive emulation mode that doesn't work so great. Remember, people buy computers for the applications that they run, and there's so much more great desktop software available for Windows than Mac that it's very hard to be a Mac user.

Sidebar What is this "API" thing?

If you're writing a program, say, a word processor, and you want to display a menu, or write a file, you have to ask the operating system to do it for you, using a very specific set of function calls which are different on every operating system. These function calls are called the API: it's the interface that an operating system, like Windows, provides to application developers, like the programmers building word processors and spreadsheets and whatnot. It's a set of thousands and thousands of detailed and fussy functions and subroutines that programmers can use, which cause the operating system to do interesting things like display a menu, read and write files, and more esoteric things like find out how to spell out a given date in Serbian, or extremely complex things like display a web page in a window. If your program uses the API calls for Windows, it's not going to work on Linux, which has different API calls. Sometimes they do approximately the same thing. That's one important reason Windows software doesn't run on Linux. If you wanted to get a Windows program to run under Linux, you'd have to reimplement the entire Windows API, which consists of thousands of complicated functions: this is almost as much work as implementing Windows itself, something which took Microsoft thousands of person-years. And if you make one tiny mistake or leave out one function that an application needs, that application will crash.

And that's why the Windows API is such an important asset to Microsoft.

(I know, I know, at this point the 2.3% of the world that uses Macintoshes are warming up their email programs to send me a scathing letter about how much they love their Macs. Once again, I'm speaking in large trends and generalizing, so don't waste your time. I know you love your Mac. I know it runs everything you need. I love you, you're a Pepper, but you're only 2.3% of the world, so this article isn't about you.)

The Two Forces at Microsoft

There are two opposing forces inside Microsoft, which I will refer to, somewhat tongue-in-cheek, as The Raymond Chen Camp and The MSDN Magazine Camp.

Raymond Chen is a developer on the Windows team at Microsoft. He's been there since 1992, and his weblog The Old New Thing is chock-full of detailed technical stories about why certain things are the way they are in Windows, even silly things, which turn out to have very good reasons.

The most impressive things to read on Raymond's weblog are the stories of the incredible efforts the Windows team has made over the years to support backwards compatibility:

Look at the scenario from the customer's standpoint. You bought programs X, Y and Z. You then upgraded to Windows XP. Your computer now crashes randomly, and program Z doesn't work at all. You're going to tell your friends, "Don't upgrade to Windows XP. It crashes randomly, and it's not compatible with program Z." Are you going to debug your system to determine that program X is causing the crashes, and that program Z doesn't work because it is using undocumented window messages? Of course not. You're going to return the Windows XP box for a refund. (You bought programs X, Y, and Z some months ago. The 30-day return policy no longer applies to them. The only thing you can return is Windows XP.)

I first heard about this from one of the developers of the hit game SimCity, who told me that there was a critical bug in his application: it used memory right after freeing it, a major no-no that happened to work OK on DOS but would not work under Windows where memory that is freed is likely to be snatched up by another running application right away. The testers on the Windows team were going through various popular applications, testing them to make sure they worked OK, but SimCity kept crashing. They reported this to the Windows developers, who disassembled SimCity, stepped through it in a debugger, found the bug, and added special code that checked if SimCity was running, and if it did, ran the memory allocator in a special mode in which you could still use memory after freeing it.

This was not an unusual case. The Windows testing team is huge and one of their most important responsibilities is guaranteeing that everyone can safely upgrade their operating system, no matter what applications they have installed, and those applications will continue to run, even if those applications do bad things or use undocumented functions or rely on buggy behavior that happens to be buggy in Windows n but is no longer buggy in Windows n+1. In fact if you poke around in the AppCompatibility section of your registry you'll see a whole list of applications that Windows treats specially, emulating various old bugs and quirky behaviors so they'll continue to work. Raymond Chen writes, "I get particularly furious when people accuse Microsoft of maliciously breaking applications during OS upgrades. If any application failed to run on Windows 95, I took it as a personal failure. I spent many sleepless nights fixing bugs in third-party programs just so they could keep running on Windows 95."

A lot of developers and engineers don't agree with this way of working. If the application did something bad, or relied on some undocumented behavior, they think, it should just break when the OS gets upgraded. The developers of the Macintosh OS at Apple have always been in this camp. It's why so few applications from the early days of the Macintosh still work. For example, a lot of developers used to try to make their Macintosh applications run faster by copying pointers out of the jump table and calling them directly instead of using the interrupt feature of the processor like they were supposed to. Even though somewhere in Inside Macintosh, Apple's official Bible of Macintosh programming, there was a tech note saying "you can't do this," they did it, and it worked, and their programs ran faster... until the next version of the OS came out and they didn't run at all. If the company that made the application went out of business (and most of them did), well, tough luck, bubby.

To contrast, I've got DOS applications that I wrote in 1983 for the very original IBM PC that still run flawlessly, thanks to the Raymond Chen Camp at Microsoft. I know, it's not just Raymond, of course: it's the whole modus operandi of the core Windows API team. But Raymond has publicized it the most through his excellent website The Old New Thing so I'll name it after him.

That's one camp. The other camp is what I'm going to call the MSDN Magazine camp, which I will name after the developer's magazine full of exciting articles about all the different ways you can shoot yourself in the foot by using esoteric combinations of Microsoft products in your own software. The MSDN Magazine Camp is always trying to convince you to use new and complicated external technology like COM+, MSMQ, MSDE, Microsoft Office, Internet Explorer and its components, MSXML, DirectX (the very latest version, please), Windows Media Player, and Sharepoint... Sharepoint! which nobody has; a veritable panoply of external dependencies each one of which is going to be a huge headache when you ship your application to a paying customer and it doesn't work right. The technical name for this is DLL Hell. It works here: why doesn't it work there?

The Raymond Chen Camp believes in making things easy for developers by making it easy to write once and run anywhere (well, on any Windows box). The MSDN Magazine Camp believes in making things easy for developers by giving them really powerful chunks of code which they can leverage, if they are willing to pay the price of incredibly complicated deployment and installation headaches, not to mention the huge learning curve. The Raymond Chen camp is all about consolidation. Please, don't make things any worse, let's just keep making what we already have still work. The MSDN Magazine Camp needs to keep churning out new gigantic pieces of technology that nobody can keep up with.

Here's why this matters.

Microsoft Lost the Backwards Compatibility Religion

Inside Microsoft, the MSDN Magazine Camp has won the battle.

The first big win was making Visual Basic.NET not backwards-compatible with VB 6.0. This was literally the first time in living memory that when you bought an upgrade to a Microsoft product, your old data (i.e. the code you had written in VB6) could not be imported perfectly and silently. It was the first time a Microsoft upgrade did not respect the work that users did using the previous version of a product.

And the sky didn't seem to fall, not inside Microsoft. VB6 developers were up in arms, but they were disappearing anyway, because most of them were corporate developers who were migrating to web development anyway. The real long term damage was hidden.

With this major victory under their belts, the MSDN Magazine Camp took over. Suddenly it was OK to change things. IIS 6.0 came out with a different threading model that broke some old applications. I was shocked to discover that our customers with Windows Server 2003 were having trouble running FogBugz. Then .NET 1.1 was not perfectly backwards compatible with 1.0. And now that the cat was out of the bag, the OS team got into the spirit and decided that instead of adding features to the Windows API, they were going to completely replace it. Instead of Win32, we are told, we should now start getting ready for WinFX: the next generation Windows API. All different. Based on .NET with managed code. XAML. Avalon. Yes, vastly superior to Win32, I admit it. But not an upgrade: a break with the past.

Outside developers, who were never particularly happy with the complexity of Windows development, have defected from the Microsoft platform en-masse and are now developing for the web. Paul Graham, who created Yahoo! Stores in the early days of the dotcom boom, summarized it eloquently: "There is all the more reason for startups to write Web-based software now, because writing desktop software has become a lot less fun. If you want to write desktop software now you do it on Microsoft's terms, calling their APIs and working around their buggy OS. And if you manage to write something that takes off, you may find that you were merely doing market research for Microsoft."

Microsoft got big enough, with too many developers, and they were too addicted to upgrade revenues, so they suddenly decided that reinventing everything was not too big a project. Heck, we can do it twice. The old Microsoft, the Microsoft of Raymond Chen, might have implemented things like Avalon, the new graphics system, as a series of DLLs that can run on any version of Windows and which could be bundled with applications that need them. There's no technical reason not to do this. But Microsoft needs to give you a reason to buy Longhorn, and what they're trying to pull off is a sea change, similar to the sea change that occurred when Windows replaced DOS. The trouble is that Longhorn is not a very big advance over Windows XP; not nearly as big as Windows was over DOS. It probably won't be compelling enough to get people to buy all new computers and applications like they did for Windows. Well, maybe it will, Microsoft certainly needs it to be, but what I've seen so far is not very convincing. A lot of the bets Microsoft made are the wrong ones. For example, WinFS, advertised as a way to make searching work by making the file system be a relational database, ignores the fact that the real way to make searching work is by making searching work. Don't make me type metadata for all my files that I can search using a query language. Just do me a favor and search the damned hard drive, quickly, for the string I typed, using full-text indexes and other technologies that were boring in 1973.

Automatic Transmissions Win the Day

Don't get me wrong... I think .NET is a great development environment and Avalon with XAML is a tremendous advance over the old way of writing GUI apps for Windows. The biggest advantage of .NET is the fact that it has automatic memory management.

A lot of us thought in the 1990s that the big battle would be between procedural and object oriented programming, and we thought that object oriented programming would provide a big boost in programmer productivity. I thought that, too. Some people still think that. It turns out we were wrong. Object oriented programming is handy dandy, but it's not really the productivity booster that was promised. The real significant productivity advance we've had in programming has been from languages which manage memory for you automatically. It can be with reference counting or garbage collection; it can be Java, Lisp, Visual Basic (even 1.0), Smalltalk, or any of a number of scripting languages. If your programming language allows you to grab a chunk of memory without thinking about how it's going to be released when you're done with it, you're using a managed-memory language, and you are going to be much more efficient than someone using a language in which you have to explicitly manage memory. Whenever you hear someone bragging about how productive their language is, they're probably getting most of that productivity from the automated memory management, even if they misattribute it.

Sidebar
Why does automatic memory management make you so much more productive? 1) Because you can write f(g(x)) without worrying about how to free the return value from g, which means you can use functions which return interesting complex data types and functions which transform interesting complex data types, in turn allowing you to work at a higher level of abstraction. 2) Because you don't have to spend any time writing code to free memory or tracking down memory leaks. 3) Because you don't have to carefully coordinate the exit points from your functions to make sure things are cleaned up properly.

Racing car aficionados will probably send me hate mail for this, but my experience has been that there is only one case, in normal driving, where a good automatic transmission is inferior to a manual transmission. Similarly in software development: in almost every case, automatic memory management is superior to manual memory management and results in far greater programmer productivity.

If you were developing desktop applications in the early years of Windows, Microsoft offered you two ways to do it: writing C code which calls the Windows API directly and managing your own memory, or using Visual Basic and getting your memory managed for you. These are the two development environments I have used the most, personally, over the last 13 years or so, and I know them inside-out, and my experience has been that Visual Basic is significantly more productive. Often I've written the same code, once in C++ calling the Windows API and once in Visual Basic, and C++ always took three or four times as much work. Why? Memory management. The easiest way to see why is to look at the documentation for any Windows API function that needs to return a string. Look closely at how much discussion there is around the concept of who allocates the memory for the string, and how you negotiate how much memory will be needed. Typically, you have to call the function twice?euro;”on the first call, you tell it that you've allocated zero bytes, and it fails with a "not enough memory allocated" message and conveniently also tells you how much memory you need to allocate. That's if you're lucky enough not to be calling a function which returns a list of strings or a whole variable-length structure. In any case, simple operations like opening a file, writing a string, and closing it using the raw Windows API can take a page of code. In Visual Basic similar operations can take three lines.

So, you've got these two programming worlds. Everyone has pretty much decided that the world of managed code is far superior to the world of unmanaged code. Visual Basic was (and probably remains) the number one bestselling language product of all time and developers preferred it over C or C++ for Windows development, although the fact that "Basic" was in the name of the product made hardcore programmers shun it even though it was a fairly modern language with a handful of object-oriented features and very little leftover gunk (line numbers and the LET statement having gone the way of the hula hoop). The other problem with VB was that deployment required shipping a VB runtime, which was a big deal for shareware distributed over modems, and, worse, let other programmers see that your application was developed in (the shame!) Visual Basic.

One Runtime To Rule Them All

And along came .NET. This was a grand project, the super-duper unifying project to clean up the whole mess once and for all. It would have memory management, of course. It would still have Visual Basic, but it would gain a new language, one which is in spirit virtually the same as Visual Basic but with the C-like syntax of curly braces and semicolons. And best of all, the new Visual Basic/C hybrid would be called Visual C#, so you would not have to tell anyone you were a "Basic" programmer any more. All those horrid Windows functions with their tails and hooks and backwards-compatibility bugs and impossible-to-figure-out string-returning semantics would be wiped out, replaced by a single clean object oriented interface that only has one kind of string. One runtime to rule them all. It was beautiful. And they pulled it off, technically. .NET is a great programming environment that manages your memory and has a rich, complete, and consistent interface to the operating system and a rich, super complete, and elegant object library for basic operations.

And yet, people aren't really using .NET much.

Oh sure, some of them are.

But the idea of unifying the mess of Visual Basic and Windows API programming by creating a completely new, ground-up programming environment with not one, not two, but three languages (or are there four?) is sort of like the idea of getting two quarreling kids to stop arguing by shouting "shut up!" louder than either of them. It only works on TV. In real life when you shout "shut up!" to two people arguing loudly you just create a louder three-way argument.

(By the way, for those of you who follow the arcane but politically-charged world of blog syndication feed formats, you can see the same thing happening over there. RSS became fragmented with several different versions, inaccurate specs and lots of political fighting, and the attempt to clean everything up by creating yet another format called Atom has resulted in several different versions of RSS plus one version of Atom, inaccurate specs and lots of political fighting. When you try to unify two opposing forces by creating a third alternative, you just end up with three opposing forces. You haven't unified anything and you haven't really fixed anything.)

So now instead of .NET unifying and simplifying, we have a big 6-way mess, with everybody trying to figure out which development strategy to use and whether they can afford to port their existing applications to .NET.

No matter how consistent Microsoft is in their marketing message ("just use .NET?euro;”trust us!"), most of their customers are still using C, C++, Visual Basic 6.0, and classic ASP, not to mention all the other development tools from other companies. And the ones that are using .NET are using ASP.NET to develop web applications, which run on a Windows server but don't require Windows clients, which is a key point I'll talk about more when I talk about the web.

Oh, Wait, There's More Coming!

Now Microsoft has so many developers cranking away that it's not enough to reinvent the entire Windows API: they have to reinvent it twice. At last year's PDC they preannounced the next major version of their operating system, codenamed Longhorn, which will contain, among other things, a completely new user interface API, codenamed Avalon, rebuilt from the ground up to take advantage of modern computers' fast display adapters and realtime 3D rendering. And if you're developing a Windows GUI app today using Microsoft's "official" latest-and-greatest Windows programming environment, WinForms, you're going to have to start over again in two years to support Longhorn and Avalon. Which explains why WinForms is completely stillborn. Hope you haven't invested too much in it. Jon Udell found a slide from Microsoft labelled "How Do I Pick Between Windows Forms and Avalon?" and asks, "Why do I have to pick between Windows Forms and Avalon?" A good question, and one to which he finds no great answer.

So you've got the Windows API, you've got VB, and now you've got .NET, in several language flavors, and don't get too attached to any of that, because we're making Avalon, you see, which will only run on the newest Microsoft operating system, which nobody will have for a loooong time. And personally I still haven't had time to learn .NET very deeply, and we haven't ported Fog Creek's two applications from classic ASP and Visual Basic 6.0 to .NET because there's no return on investment for us. None. It's just Fire and Motion as far as I'm concerned: Microsoft would love for me to stop adding new features to our bug tracking software and content management software and instead waste a few months porting it to another programming environment, something which will not benefit a single customer and therefore will not gain us one additional sale, and therefore which is a complete waste of several months, which is great for Microsoft, because they have content management software and bug tracking software, too, so they'd like nothing better than for me to waste time spinning cycles catching up with the flavor du jour, and then waste another year or two doing an Avalon version, too, while they add features to their own competitive software. Riiiight.

No developer with a day job has time to keep up with all the new development tools coming out of Redmond, if only because there are too many dang employees at Microsoft making development tools!

It's Not 1990

Microsoft grew up during the 1980s and 1990s, when the growth in personal computers was so dramatic that every year there were more new computers sold than the entire installed base. That meant that if you made a product that only worked on new computers, within a year or two it could take over the world even if nobody switched to your product. That was one of the reasons Word and Excel displaced WordPerfect and Lotus so thoroughly: Microsoft just waited for the next big wave of hardware upgrades and sold Windows, Word and Excel to corporations buying their next round of desktop computers (in some cases their first round). So in many ways Microsoft never needed to learn how to get an installed base to switch from product N to product N+1. When people get new computers, they're happy to get all the latest Microsoft stuff on the new computer, but they're far less likely to upgrade. This didn't matter when the PC industry was growing like wildfire, but now that the world is saturated with PCs most of which are Just Fine, Thank You, Microsoft is suddenly realizing that it takes much longer for the latest thing to get out there. When they tried to "End Of Life" Windows 98, it turned out there were still so many people using it they had to promise to support that old creaking grandma for a few more years.

Unfortunately, these Brave New Strategies, things like .NET and Longhorn and Avalon, trying to create a new API to lock people into, can't work very well if everybody is still using their good-enough computers from 1998. Even if Longhorn ships when it's supposed to, in 2006, which I don't believe for a minute, it will take a couple of years before enough people have it that it's even worth considering as a development platform. Developers, developers, developers, and developers are not buying into Microsoft's multiple-personality-disordered suggestions for how we should develop software.

Enter the Web

I'm not sure how I managed to get this far without mentioning the Web. Every developer has a choice to make when they plan a new software application: they can build it for the web or they can build a "rich client" application that runs on PCs. The basic pros and cons are simple: Web applications are easier to deploy, while rich clients offer faster response time enabling much more interesting user interfaces.

Web Applications are easier to deploy because there's no installation involved. Installing a web application means typing a URL in the address bar. Today I installed Google's new email application by typing Alt+D, gmail, Ctrl+Enter. There are far fewer compatibility problems and problems coexisting with other software. Every user of your product is using the same version so you never have to support a mix of old versions. You can use any programming environment you want because you only have to get it up and running on your own server. Your application is automatically available at virtually every reasonable computer on the planet. Your customers' data, too, is automatically available at virtually every reasonable computer on the planet.

But there's a price to pay in the smoothness of the user interface. Here are a few examples of things you can't really do well in a web application:

  1. Create a fast drawing program
  2. Build a real-time spell checker with wavy red underlines
  3. Warn users that they are going to lose their work if they hit the close box of the browser
  4. Update a small part of the display based on a change that the user makes without a full roundtrip to the server
  5. Create a fast keyboard-driven interface that doesn't require the mouse
  6. Let people continue working when they are not connected to the Internet

These are not all big issues. Some of them will be solved very soon by witty Javascript developers. Two new web applications, Gmail and Oddpost, both email apps, do a really decent job of working around or completely solving some of these issues. And users don't seem to care about the little UI glitches and slowness of web interfaces. Almost all the normal people I know are perfectly happy with web-based email, for some reason, no matter how much I try to convince them that the rich client is, uh, richer.

So the Web user interface is about 80% there, and even without new web browsers we can probably get 95% there. This is Good Enough for most people and it's certainly good enough for developers, who have voted to develop almost every significant new application as a web application.

Which means, suddenly, Microsoft's API doesn't matter so much. Web applications don't require Windows.

It's not that Microsoft didn't notice this was happening. Of course they did, and when the implications became clear, they slammed on the brakes. Promising new technologies like HTAs and DHTML were stopped in their tracks. The Internet Explorer team seems to have disappeared; they have been completely missing in action for several years. There's no way Microsoft is going to allow DHTML to get any better than it already is: it's just too dangerous to their core business, the rich client. The big meme at Microsoft these days is: "Microsoft is betting the company on the rich client." You'll see that somewhere in every slide presentation about Longhorn. Joe Beda, from the Avalon team, says that "Avalon, and Longhorn in general, is Microsoft's stake in the ground, saying that we believe power on your desktop, locally sitting there doing cool stuff, is here to stay. We're investing on the desktop, we think it's a good place to be, and we hope we're going to start a wave of excitement..."

The trouble is: it's too late.

I'm a Little Bit Sad About This, Myself

I'm actually a little bit sad about this, myself. To me the Web is great but Web-based applications with their sucky, high-latency, inconsistent user interfaces are a huge step backwards in daily usability. I love my rich client applications and would go nuts if I had to use web versions of the applications I use daily: Visual Studio, CityDesk, Outlook, Corel PhotoPaint, QuickBooks. But that's what developers are going to give us. Nobody (by which, again, I mean "fewer than 10,000,000 people") wants to develop for the Windows API any more. Venture Capitalists won't invest in Windows applications because they're so afraid of competition from Microsoft. And most users don't seem to care about crappy Web UIs as much as I do.

And here's the clincher: I noticed (and confirmed this with a recruiter friend) that Windows API programmers here in New York City who know C++ and COM programming earn about $130,000 a year, while typical Web programmers using managed code languages (Java, PHP, Perl, even ASP.NET) earn about $80,000 a year. That's a huge difference, and when I talked to some friends from Microsoft Consulting Services about this they admitted that Microsoft had lost a whole generation of developers. The reason it takes $130,000 to hire someone with COM experience is because nobody bothered learning COM programming in the last eight years or so, so you have to find somebody really senior, usually they're already in management, and convince them to take a job as a grunt programmer, dealing with (God help me) marshalling and monikers and apartment threading and aggregates and tearoffs and a million other things that, basically, only Don Box ever understood, and even Don Box can't bear to look at them any more.

Much as I hate to say it, a huge chunk of developers have long since moved to the web and refuse to move back. Most .NET developers are ASP.NET developers, developing for Microsoft's web server. ASP.NET is brilliant; I've been working with web development for ten years and it's really just a generation ahead of everything out there. But it's a server technology, so clients can use any kind of desktop they want. And it runs pretty well under Linux using Mono.

None of this bodes well for Microsoft and the profits it enjoyed thanks to its API power. The new API is HTML, and the new winners in the application development marketplace will be the people who can make HTML sing.


Human Task Switches Considered Harmful

Monday, February 12, 2001

When you're managing a team of programmers, one of the first things you have to learn to get right is task allocation. That's just a five-dollar word for giving people things to do. It's known colloquially as "file dumping" in Hebrew (because you dump files in peoples' laps). And how you decide which files to dump in which laps is one of the areas where you can get incredible productivity benefits if you do it right. Do it wrong, and you can create one of those gnarly situations where nobody gets anything accomplished and everybody complains that "nothing ever gets done around here."

Since this site is for programmers, I'm going to warm up your brains a little bit with a programming problem.

Suppose you have two separate computations to perform, A and B. Each computation requires 10 seconds of CPU time. You have one CPU which, for the sake of this problem, doesn't have anything else in the queue.

On our CPU, multitasking is optional. So you can either do these computations one after the other...

Sequential Processing

Computation A |  |  |  |  |  |  |  |  |  | Computation B
1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | 10 | 11 | 12 | 13 | 14 | 15 | 16 | 17 | 18 | 19 | 20

... or, you can multitask. If you multitask, on this particular CPU, tasks run for 1 second at a time, and a task switch takes no time at all.

Multitasking

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20

Which would you rather do? Most people's gut reaction is that multitasking is better. In both cases, you have to wait 20 seconds to get both of your answers. But think about how long it takes to get the results to each computation.

In both cases, the results of Computation B (shown in blue) take 20 seconds to arrive. But look at Computation A. With multitasking, its results take 19 seconds to arrive... yet with sequential processing they are ready in only 10 seconds.

In other words, in this nice contrived example, the average time per computation is lower (15 seconds rather than 19.5 seconds) when you do sequential processing rather than multitasking. (Actually, it's not such a contrived example -- it's based on a real problem Jared had to solve at work).

Method | Computation A takes | Computation B takes | Average
Sequential | 10 seconds | 20 seconds | 15
Multitasking | 19 seconds | 20 seconds | 19.5

Earlier I said that "a task switch takes no time at all." Actually, on real CPUs, a task switch takes a little bit of time... basically enough time to save out the state of the CPU registers and load the CPU registers for the other task. Realistically, this is as close to negligible as possible. But to make life interesting, let's imagine that task switches take half a second. Now things look even worse:

Method | Computation A takes | Computation B takes | Average
Sequential | 10 seconds | 20 + 1 task switch = 20.5 seconds | 15.25
Multitasking | 19 + 18 task switches = 28 seconds | 20 + 19 task switches = 29.5 seconds | 28.75

Now ... just humor me, I know this is silly ... what if task switches take a whole minute?

Method | Computation A takes | Computation B takes | Average
Sequential | 10 seconds | 20 + 1 task switch = 80 seconds | 45 seconds
Multitasking | 19 + 18 task switches = 1099 seconds | 20 + 19 task switches = 1160 seconds | almost 19 minutes!

The longer a task switch takes, the worse the multitasking penalty.

That, in and of itself, is not so earth shaking, is it? Pretty soon I'm going to be getting irate email from morons accusing me of being "against" multitasking. "Do you want to go back to the days of DOS when you had to exit WordPerfect to run 1-2-3?" they will ask me.

But that's not my point. I just want you to agree with me that in this kind of example:

a) sequential processing gets you results faster on average, and

b) the longer it takes to task switch, the bigger the penalty you pay for multitasking.

OK, back to the more interesting topic of managing humans, not CPUs. The trick here is that when you manage programmers, specifically, task switches take a really, really, really long time. That's because programming is the kind of task where you have to keep a lot of things in your head at once. The more things you remember at once, the more productive you are at programming. A programmer coding at full throttle is keeping zillions of things in their head at once: everything from names of variables, data structures, important APIs, the names of utility functions that they wrote and call a lot, even the name of the subdirectory where they store their source code. If you send that programmer to Crete for a three week vacation, they will forget it all. The human brain seems to move it out of short-term RAM and swaps it out onto a backup tape where it takes forever to retrieve.

How long? Well, my software company recently dropped what we were doing (developing a software product codenamed CityDesk) to help a client with a bit of an emergency situation for three weeks. When we got back to the office, it seemed to take another three weeks to get back to full speed on CityDesk.

On the individual level -- have you ever noticed that you can assign one job to one person, and they'll do a great job, but if you assign two jobs to that person, they won't really get anything done? They'll either do one job well and neglect the other, or they'll do both jobs so slowly you feel like slugs have more zip. That's because programming tasks take so long to task switch. I feel like when I have two programming projects on my plate at once, the task switch time is something like 6 hours. In an 8-hour day, that means multitasking reduces my productivity to 2 hours per day. Pretty dismal.

As it turns out, if you give somebody two things to work on, you should be grateful if they "starve" one task and only work on one, because they're going to get more stuff done and finish the average task sooner. In fact, the real lesson from all this is that you should never let people work on more than one thing at once. Make sure they know what it is. Good managers see their responsibility as removing obstacles so that people can focus on one thing and really get it done. When emergencies come up, think about whether you can handle it yourself before you delegate it to a programmer who is deeply submersed in a project.


In Defense of Not-Invented-Here Syndrome

Sunday, October 14, 2001

Time for a pop quiz.

1. Code Reuse is:

a) Good
b) Bad

2. Reinventing the Wheel is:

a) Good
b) Bad

3. The Not-Invented-Here Syndrome is:

a) Good
b) Bad

Of course, everybody knows that you should always leverage other people's work. The correct answers are, of course, 1(a) 2(b) 3(b).

Right?

Not so fast, there!

The Not-Invented-Here Syndrome is considered a classic management pathology, in which a team refuses to use a technology that they didn't create themselves. People with NIH syndrome are obviously just being petty, refusing to do what's in the best interest of the overall organization because they can't find a way to take credit. (Right?) The Boring Business History Section at your local megabookstore is rife with stories about stupid teams that spend millions of dollars and twelve years building something they could have bought at Egghead for $9.99. And everybody who has paid any attention whatsoever to three decades of progress in computer programming knows that Reuse is the Holy Grail of all modern programming systems.

Right. Well, that's what I thought, too. So when I was the program manager in charge of the first implementation of Visual Basic for Applications, I put together a careful coalition of four, count them, four different teams at Microsoft to get custom dialog boxes in Excel VBA. The idea was complicated and fraught with interdependencies. There was a team called AFX that was working on some kind of dialog editor. Then we would use this brand new code from the OLE group which let you embed one app inside another. And the Visual Basic team would provide the programming language behind it. After a week of negotiation I got the AFX, OLE, and VB teams to agree to this in principle.

I stopped by Andrew Kwatinetz's office. He was my manager at the time and taught me everything I know. "The Excel development team will never accept it," he said. "You know their motto? 'Find the dependencies -- and eliminate them.' They'll never go for something with so many dependencies."

In-ter-est-ing. I hadn't known that. I guess that explained why Excel had its own C compiler.

By now I'm sure many of my readers are rolling on the floor laughing. "Isn't Microsoft stupid," you're thinking, "they refused to use other people's code and they even had their own compiler just for one product."

Not so fast, big boy! The Excel team's ruggedly independent mentality also meant that they always shipped on time, their code was of uniformly high quality, and they had a compiler which, back in the 1980s, generated pcode and could therefore run unmodified on Macintosh's 68000 chip as well as Intel PCs. The pcode also made the executable file about half the size that Intel binaries would have been, which loaded faster from floppy disks and required less RAM.

"Find the dependencies -- and eliminate them." When you're working on a really, really good team with great programmers, everybody else's code, frankly, is bug-infested garbage, and nobody else knows how to ship on time. When you're a cordon bleu chef and you need fresh lavender, you grow it yourself instead of buying it in the farmers' market, because sometimes they don't have fresh lavender or they have old lavender which they pass off as fresh.

Indeed during the recent dotcom mania a bunch of quack business writers suggested that the company of the future would be totally virtual -- just a trendy couple sipping Chardonnay in their living room outsourcing everything. What these hyperventilating "visionaries" overlooked is that the market pays for value added. Two yuppies in a living room buying an e-commerce engine from company A and selling merchandise made by company B and warehoused and shipped by company C, with customer service from company D, isn't honestly adding much value. In fact, if you've ever had to outsource a critical business function, you realize that outsourcing is hell. Without direct control over customer service, you're going to get nightmarishly bad customer service -- the kind people write about in their weblogs when they tried to get someone, anyone, from some phone company to do even the most basic thing. If you outsource fulfillment, and your fulfillment partner has a different idea about what constitutes prompt delivery, your customers are not going to be happy, and there's nothing you can do about it, because it took 3 months to find a fulfillment partner in the first place, and in fact, you won't even know that your customers are unhappy, because they can't talk to you, because you've set up an outsourced customer service center with the explicit aim of not listening to your own customers. That e-commerce engine you bought? There's no way it's going to be as flexible as what Amazon does with obidos, which they wrote themselves. (And if it is, then Amazon has no advantage over their competitors who bought the same thing). And no off-the-shelf web server is going to be as blazingly fast as what Google does with their hand-coded, hand-optimized server.

This principle, unfortunately, seems to be directly in conflict with the ideal of "code reuse good -- reinventing wheel bad."

The best advice I can offer:

If it's a core business function -- do it yourself, no matter what.

Pick your core business competencies and goals, and do those in house. If you're a software company, writing excellent code is how you're going to succeed. Go ahead and outsource the company cafeteria and the CD-ROM duplication. If you're a pharmaceutical company, write software for drug research, but don't write your own accounting package. If you're a web accounting service, write your own accounting package, but don't try to create your own magazine ads. If you have customers, never outsource customer service.

If you're developing a computer game where the plot is your competitive advantage, it's OK to use a third party 3D library. But if cool 3D effects are going to be your distinguishing feature, you had better roll your own.

The only exception to this rule, I suspect, is if your own people are more incompetent than everyone else, so whenever you try to do anything in house, it's botched up. Yes, there are plenty of places like this. If you're in one of them, I can't help you.

Discuss


Incentive Pay Considered Harmful

Monday, April 03, 2000

Mike Murray, a surprisingly hapless HR manager at Microsoft, made a number of goofs, but the doozie was introducing a "Ship It" award shortly after he started the job. The idea was that you would get a big lucite tombstone the size of a dictionary when your product shipped. This was somehow supposed to give you an incentive to work, you see, because if you didn't do your job-- no lucite for you! Makes you wonder how Microsoft ever shipped software before the lucite slab.

The Ship It program was announced with an incredible amount of internal fanfare and hoopla at a big company picnic. For weeks before the event, teaser posters appeared all over the corporate campus with a picture of Bill Gates and the words "Why is this man smiling?" I'm not sure what that meant. Was Bill smiling because he was happy that we now had an incentive to ship software? But at the company picnic, it became apparent that the employees felt like they were being treated like children. There was a lot of booing. The Excel programming team held up a huge sign that said "Why is the Excel team yawning?" The Ship It award was so despised that there is even a (non-fiction) episode in Douglas Coupland's classic Microserfs in which a group of programmers try to destroy one with a blow torch.

Treating your rocket scientist employees as if they were still in kindergarten is not an isolated phenomenon. Almost every company has some kind of incentive program that is insulting and demeaning.

At two of the companies I've worked for, the most stressful time of year was the twice-yearly performance review period. For some reason, the Juno HR department and the Microsoft HR department must have copied their performance review system out of the same Dilbertesque management book, because both programs worked exactly the same way. First, you gave "anonymous" upward reviews for your direct manager (as if that could be done in an honest way). Then, you filled out optional "self-evaluation" forms, which your manager "took into account" in preparing your performance review. Finally, you got a numerical score, in lots of non-scalar categories like "works well with others", from 1-5, where the only possible scores were actually 3 or 4. Managers submitted bonus recommendations upwards, which were completely ignored and everybody received bonuses that were almost completely random. The system never took into account the fact that people have different and unique talents, all of which are needed for a team to work well.

Performance reviews were stressful for a couple of reasons. Many of my friends, especially the ones whose talents were very significant but didn't show up on the traditional scales, tended to get lousy performance reviews. For example, one friend of mine was a cheerful catalyst, a bouncy cruise director who motivated everyone else when the going got tough. He was the glue that held his team together. But he tended to get negative reviews, because his manager didn't understand his contribution. Another friend was incredibly insightful strategically; his conversations with other people about how things should be done allowed everyone else to do much better work. He tended to spend more time than average trying out new technologies; in this area he was invaluable to the rest of the team. But in terms of lines of code, he wrote less than average, and his manager was too stupid to notice all his other contributions, so he always got negative reviews, too. Negative reviews, obviously, have a devastating effect on morale. In fact, giving somebody a review that is positive, but not as positive as that person expected, also has a negative effect on morale.

The effect of reviews on morale is lopsided: while negative reviews hurt morale a lot, positive reviews have no effect on morale or productivity. The people who get them are already working productively. For them, a positive review makes them feel like they are doing good work in order to get the positive review... as if they were Pavlovian dogs working for a treat, instead of professionals who actually care about the quality of the work that they do.

And herein lies the rub. Most people think that they do pretty good work (even if they don't). It's just a little trick our minds play on us to keep life bearable. So if everybody thinks they do good work, and the reviews are merely correct (which is not very easy to achieve), then most people will be disappointed by their reviews. The cost of this in morale is hard to understate. On teams where performance reviews are done honestly, they tend to result in a week or so of depressed morale, moping, and some resignations. They tend to drive wedges between team members, often because the poorly-rated are jealous of the highly-rated, in a process that DeMarco and Lister call teamicide: the inadvertent destruction of jelled teams.

Alfie Kohn, in a now-classic Harvard Business Review article, wrote:

... at least two dozen studies over the last three decades have conclusively shown that people who expect to receive a reward for completing a task or for doing that task successfully simply do not perform as well as those who expect no reward at all. [HBR Sept/Oct 93]

He concludes that "incentives (or bribes) simply can't work in the workplace". DeMarco and Lister go further, stating unequivocally that any kind of workplace competition, any scheme of rewards and punishments, and even the old fashion trick of "catching people doing something right and rewarding them," all do more harm than good. Giving somebody positive reinforcement (such as stupid company ceremonies where people get plaques) implies that they only did it for the lucite plaque; it implies that they are not independent enough to work unless they are going to get a cookie; and it's insulting and demeaning.

Most software managers have no choice but to go along with performance review systems that are already in place. If you're in this position, the only way to prevent teamicide is to simply give everyone on your team a gushing review. But if you do have any choice in the matter, I'd recommend that you run fleeing from any kind of performance review, incentive bonus, or stupid corporate employee-of-the-month program.


International Readers

Tuesday, November 14, 2000

Joel on Software has subscribers in 44 countries:

Argentina
Australia
Austria
Belgium
Botswana
Brasil
Canada
Chile
Denmark
Estonia
Finland
France
Germany
Greece
Hong Kong
Iceland
India
Ireland
Israel
Italy
Latvia
Lebanon
Lithuania
Malaysia
Mexico
Netherlands
New Zealand
Norway
Philippines
Poland
Portugal
Romania
Russia
Singapore
Slovak Republic
Slovenia
South Africa
Spain
Sweden
UK
United Arab Emirates
USA
Western Samoa
Yugoslavia


It's Not Just Usability

Monday, September 06, 2004

For years and years, self-fashioned pundits, like, uh, me, have been nattering endlessly about usability, and how important it is to make software usable. Jakob Nielsen has a mathematical formula he'll reveal to you in exchange for $122 which you can use to calculate the value of usability. (If the expected value of usability is greater than $122, I guess you make a profit.)

I have a book you can buy for a lot less that tells you some of the principles of designing usable software, but there's no math involved, and you'll be out the price of the book.

In that book, on page 31, I showed an example from what was, at the time, the most popular software application on Earth, Napster. The main Napster window used buttons to flip between the five screens. Due to a principle in usability called

And yet, Napster was the most popular software application on Earth.

In an early version of the manuscript, I actually wrote something like "this just goes to show you that usability ain't all that important," which was an odd thing to be writing in a book about usability. I was sort of relieved when the typesetter announced that I had to shorten that paragraph. So I deleted the sentence.

But there's a scary element of truth to it—scary to UI professionals, at least: an application that does something really great that people really want to do can be pathetically unusable, and it will still be a hit. And an application can be the easiest thing in the world to use, but if it doesn't do anything anybody wants, it will flop. UI consultants are constantly on the defensive, working up improbable ROI formulas about the return on investment clients will get from their $75,000 usability project, precisely because usability is perceived as "optional," and the scary thing is, in a lot of cases, it is. In a lot of cases. The CNN website has nothing to be gained from a usability consultant. I'll go out on a limb and say that there is not a single content-based website online that would gain even one dollar in revenue by improving usability, because content-based websites (by which I mean, websites that are not also applications) are already so damn usable.

Anyway.

My goal today is not to whine about how usability is not important... usability is important at the margins, and there a lots of examples where bad usability kills people in small planes, creates famine and pestilence, etc.

My goal today is to talk about the next level of software design issues, after you've got the UI right: designing the social interface.

I need to explain that, I guess.

Software in the 1980s, when usability was "invented," was all about computer-human interaction. A lot of software still is. But the Internet brings us a new kind of software: software that's about human-human interaction.

Discussion groups. Social networking. Online classifieds. Oh, and, uh, email. It's all software that mediates between people, not between the human and the computer.

When you're writing software that mediates between people, after you get the usability right, you have to get the social interface right. And the social interface is

The best way to illustrate social interfaces is with a few examples of failures and successes.

Some Examples

First, a failing social interface. Every week I get an email from somebody I've never heard of asking me to become a part of his or her social network. I usually don't know the person, so I feel a little bit miffed and delete the message. Someone told me why this happens: one of those social networking software companies has a tool that goes through your email address book and sends email to everyone asking them to join in. Now, combine this with the feature that some email software saves the sender's address of every incoming message, and the feature that when you go to sign up for the Joel on Software email bulletin you get a confirmation message asking if you really want to join, and voila: all kinds of people who I don't know are running software that is inadvertently asking me to confirm that I'm their friend. Thank you for subscribing to my newsletter, but no, I'm not going to introduce you to Bill Gates. I currently have a policy of not joining any of these social networks, because they strike me as going strongly against the grain of how human networks really work.

Now, let's look at a successful social interface. Many humans are less inhibited when they're typing than when they are speaking face-to-face. Teenagers are less shy. With cellphone text messages, they're more likely to ask each other out on dates. That genre of software was so successful socially that it's radically improving millions of people's love lives (or at least their social calendars). Even though text messaging has a ghastly user interface, it became extremely popular with the kids. The joke of it is that there's a much better user interface built into every cellphone for human to human communication: this clever thing called "phone calls." You dial a number after which everything you say can be heard by the other person, and vice versa. It's that simple. But it's not as popular in some circles as this awkward system where you break your thumbs typing huge strings of numbers just to say "damn you're hot," because that string of numbers gets you a date, and you would never have the guts to say "damn you're hot" using your larynx.

Another social software success is ebay. When I first heard about ebay, I said, "Nonsense! That will never work. Nobody's going to send money to some random person they encountered on the Internet in hopes that person will out of the goodness of their hearts actually ship them some merchandise." A lot of people thought this. We were all wrong. Wrong, wrong, wrong. Ebay made a big bet on the cultural anthropology of human beings and won. The great thing about ebay is that it was a huge success precisely because it seemed like a terrible idea at the time, and so nobody else tried it, until ebay locked in the network effects and first-mover advantage.

In addition to absolute success and failures in social software, there are also social software side effects. The way social software behaves determines a huge amount about the type of community that develops. Usenet clients have this big-R command which is used to reply to a message while quoting the original message with those elegant >'s in the left column. And the early newsreaders were not threaded, so if you wanted to respond to someone's point coherently, you had to quote them using the big-R feature. This led to a particularly Usenet style of responding to an argument: the line-by-line nitpick. It's fun for the nitpicker but never worth reading. (By the way, the political bloggers, newcomers to the Internet, have reinvented this technique, thinking they were discovering something fun and new, and called it fisking, for reasons I won't go into. Don't worry, it's not dirty.) Even though human beings had been debating for centuries, a tiny feature of a software product produced a whole new style of debating.

Small changes in software can make big differences in the way that software supports, or fails to support, its social goals. Danah Boyd has a great critique of social software networks, Autistic Social Software, blasting the current generation of this software for forcing people to behave autistically:

Consider, for a moment, the recent surge of interest in articulated social networks such as Friendster, Tribe, LinkedIn, Orkut and the like. These technologies attempt to formalize how people should construct and manage their relationships. They assume that you can rate your friends. In some cases, they procedurally direct how people can engage with new people by giving you an absolute process through which you can contact others.

While this approach certainly has its merits because it is computationally possible, i'm terrified when people think that this models social life. It's so simplistic that people are forced to engage as though they have autism, as though they must interact procedurally. This approach certainly aids people who need that kind of systematization, but it is not a model that universally makes sense. Furthermore, what are the implications of having technology prescribe mechanistic engagement? Do we really want a social life that encourages autistic interactions?

When software implements social interfaces while disregarding cultural anthropology, it's creepy and awkward and doesn't really work.

Designing Social Software

Let me give you an example of social interface design.

Suppose your user does something they shouldn't have done.

Good usability design says that you should tell them what they did wrong, and tell them how to correct it. Usability consultants are marketing this under the brand name "Defensive Design."

When you're working on social software, this is too naive.

Maybe the thing that they did wrong was to post an advertisement for Viagra on a discussion group.

Now you tell them, "Sorry, Viagra is not a suitable topic. Your post has been rejected."

Guess what they'll do? They'll post an advertisement for V1agra. (Either that, or they'll launch into a long boring rant about censorship and the First Amendment.)

With social interface engineering, you have to look at sociology and anthropology. In societies, there are freeloaders, scammers, and other miscreants. In social software, there will be people who try to abuse the software for their own profit at the expense of the rest of the society. Unchecked, this leads to something economists call the tragedy of the commons.

Whereas the goal of user interface design is to help the user succeed, the goal of social interface design is to help the society succeed, even if it means one user has to fail.

So a good social interface designer might say, let's not display an error message. Let's just pretend that the post about Viagra was accepted. Show it to the original poster, so he feels smug and moves on to the next inappropriate discussion group. But don't show it to anyone else.

Indeed one of the best ways to deflect attacks is to make it look like they're succeeding. It's the software equivalent of playing dead.

No, it doesn't work 100% of the time. It works 95% of the time, and it reduces the problems you'll have twenty-fold. Like everything else in sociology, it's a fuzzy heuristic. It kind of works a lot of the time, so it's worth doing, even if it's not foolproof. The Russian mafia with their phishing schemes will eventually work around it. The idiot Floridians in trailer parks trying to get rich quick will move on. 90% of the spam I get today is still so hopelessly naive about spam filters that it would even get caught by the pathetic junk filter built into Microsoft Outlook, and you've got to have really lame spam to get caught by that scrawny smattering of simplistic searchphrases.

Marketing Social Interfaces

A few months ago I realized that a common theme in the software we've built at Fog Creek is an almost obsessive attention to getting the social interfaces right. For example, FogBugz has lots of features, and even more non-features, designed to make bug tracking actually happen. Time and time again customers tell me that their old bug tracking software was never getting used, because it did not align with the way people wanted to work together, but when they rolled out FogBugz, people actually starting using it, and became addicted to it, and it changed the way they worked together. I know that FogBugz works because we have a very high upgrade rate when there's a new version, which tells me FogBugz is not just shelfware, and because even customers who buy huge blocks of licenses keep coming back for more user licenses as the product spreads around their organization and really gets used. This is something I'm really proud of. Software used in teams usually fails to take hold, because it requires everyone on the team to change the way they work simultaneously, something which anthropologists will tell you is vanishingly unlikely. For that reason FogBugz has lots of design decisions which make it useful even for a single person on a team, and lots of design features which encourage it to spread to other members of the team gradually until everyone is using it.

The discussion group software used on this site, which will soon be for sale as a feature of FogBugz 4.0, is even more obsessed with getting the social interface aspects exactly right. There are dozens of features and non-features and design decisions which collectively lead to a very high level of interesting conversation with the best signal-to-noise ratio of any discussion group I've ever participated in. I wrote a lot about this in my article Building Communities with Software.

Since then, I've become even more devoted to the idea of the value of good social interface design: we bring in experts like Clay Shirky (a pioneer in the field), we do bold experiments on the poor citizens of the Joel on Software discussion group (many of which are so subtle as to be virtually unnoticeable, for example, the fact that we don't show you the post you're replying to while you type your reply in hopes of cutting down quoting, which makes it easier to read a thread), and we're investing heavily in advanced algorithms to reduce discussion group spam.

A New Field

Social interface design is still a field in its infancy. I'm not aware of any books on the subject; there are only a few people working in the research side of the field, and there's no organized science of social interface design. In the early days of usability design, software companies recruited ergonomics experts and human factors experts to help design usable products. Ergonomics experts knew a lot about the right height for a desk, but they didn't know how to design GUIs for file systems, so a new field arose. Eventually the new discipline of user interface design came into its own, and figured out the concepts like consistency, affordability, feedback, etc., which became the cornerstone of the science of UI design.

Over the next decade, I expect that software companies will hire people trained as anthropologists and ethnographers to work on social interface design. Instead of building usability labs, they'll go out into the field and write ethnographies. And hopefully, we'll figure out the new principles of social interface design. It's going to be fascinating... as fun as user interface design was in the 1980s... so stay tuned.

Discuss this topic on the Joel on Software Social Interface Design Forum.


Juggling Tasks in Excel

Monday, May 08, 2000

In response to Painless Software Schedules, Chris Sells writes:

I have some questions about using Excel, though. While I'm no fan of MS Project, once I figured it out (about 8 hours worth of banging my head on the wall and 2 hours on the phone with MS tech support), it was able to do some fun things that I haven't figured out how to do with Excel, noteably calculate dates and critical paths for multiple people. Once I had the raw data entered, it was easy to juggle tasks from person to person and than rebalance until everyone was going to finish at roughly the same time. How can I use Excel to do that?

My response to Chris:

As for rejuggling tasks -- if you have to do this at anything other than the most macro level, chances are that juggling will wind up actually changing the duration of the task in a dangerous way. For example, if you asked me to write code for an EXE that connected to a web site, checked if a new version of the EXE was available, and if so, downloaded it and replaced the EXE on the hard drive (even while the EXE was running), I would estimate that at "1 day" because I wrote code like that two weeks ago and it took me 1 day. Among other things, I know how to use the WININET functions for reading a web page; I've done stuff like this in the past so I know to wrap the version number result from the server in XML so some clients don't get "version 404" as the result; I know how to replace an EXE on the fly and have code snippets to do this.
On the other hand, assign that task to almost anybody else and it will almost certainly take 3 or 4 times as long, even for a good programmer, because their particular set of 'memorized' skills is probably radically different. If you asked Don Box to do some bizzaro thing with ATL and apartment model COM threads that uses MTS, he would do it in 20 minutes. It would take me 2 weeks, and I would cry a lot during the process.
Now... you still sure you want to reassign tasks *after* a programmer has estimated it (based on their own particular skill set?)
But: to answer the actual question you asked, the way you do it with Excel is to make a column called "Person" with the person the task is assigned to, then use the SUMIF function to make a quick table of hours-per-person, and "rejuggle" until they're close. But reality check: unless you're doing particularly boring work that your team has all done millions of times before (teaching COM classes? ;) it ain't a gonna work.

Advertainment! Do you need to control a computer remotely, even when firewalls get in the way? My company's latest product, Fog Creek Copilot, is a remote control system that requires no setup, no configuration, and works even if both users are behind firewalls. It's designed to make remote tech support easy.

Enter your email address to receive a (very occasional) email whenever I write a major new article. You can unsubscribe at any time, of course.
Email:


Let's Take Sabbaticals!

Friday, December 24, 1999

I'm on sabbatical now. I left my last job, at Juno Online Services, last November. Since then I've been travelling, learning, and resting up for the next big thing.

Over the next few months I plan to write several articles about software development in this weblog. Keep an eye out for discussions on managing teams of developers at high tech companies; how to motivate and excite people; strategies for software firms, and more.


Lord Palmerston on Programming

Wednesday, December 11, 2002

There was a time when if you read one book by Peter Norton, you literally knew everything there was to know about programming the IBM-PC. Over the last 20 years, programmers around the world have been hard at work building abstraction upon abstraction on top of the IBM-PC to make it easier to program and more powerful.

But the law of leaky abstractions means that even as they built the abstractions that are supposed to make programming easier, the sheer amount of stuff you have to know to be a great programmer is expanding all the time.

Becoming proficient, really proficient, in just one programming world takes years. Sure, lots of bright teenagers learn Delphi one week and Python the next week and Perl the next week and think they are proficient. Yet they don't have the foggiest clue how much they're missing.

I've been working with ASP and VBScript since it first came out. VBScript is the dinkiest language on earth and ASP programming consists of learning about 5 classes, only two of which you use very often. And only now do I finally feel like I know the best way to architect an ASP/VBScript application. I finally think I know where the best place to put database access code is, the best way to use ADO to get recordsets, the best way to separate HTML and code, etc. And I finally use regexps instead of one-off string manipulation functions. Only last week, I learned how to get COM objects out of memory so you can recompile them (without restarting the whole web server).

Fog Creek is too small to have specialists, so when I needed to write a really good installer for FogBUGZ, our ASP/VBScript based product, I drew on several years of C++/MFC experience, and years of experience with Windows APIs, and good Corel PhotoPaint skills to create a neat picture in the corner of the wizard. Then to get FogBUGZ to work perfectly with Unicode, I had to write a little ActiveX control using C++ and ATL, which drew upon years of C++ and COM experience and a week or so learning about character encodings when I implemented that code in CityDesk.

So when we had a weird NT 4.0-only bug, it took me 3 minutes to debug, because I knew how to use VMWare, and I had a clean NT 4.0 machine set up in VMWare, and I knew how to do remote debugging with Visual C++, and I knew to look in the EAX register to get the return value from a function. Someone who was new to this all might have taken an hour or more to debug the same problem, but I already knew a tremendous amount of "stuff" that I've been learning, basically, since 1982 when I got my first IBM-PC and that Norton book.

Leaky abstractions mean that we live with a hockey stick learning curve: you can learn 90% of what you use day by day with a week of learning. But the other 10% might take you a couple of years catching up. That's where the really experienced programmers will shine over the people who say "whatever you want me to do, I can just pick up the book and learn how to do it." If you're building a team, it's OK to have a lot of less experienced programmers cranking out big blocks of code using the abstract tools, but the team is not going to work if you don't have some really experienced members to do the really hard stuff.

There are a lot of programming worlds, each of which requires a tremendous amount of knowledge for real proficiency. Here are the three I personally know best:

  • MFC/C++/Windows
  • VBScript/ASP
  • Visual Basic

All, basically, what you would call Windows programming. Yes, I've written Unix code and Java code, but not very much. My proficiency in Windows programming comes from knowing not just the basic technologies but also the whole supporting infrastructure. So, I claim, I'm really good at Windows programming because I also know COM, ATL, C++, 80x86 Assembler, Windows APIs, IDispatch (OLE Automation), HTML, the DOM, the Internet Explorer object model, Windows NT and Windows 95 internals, LAN Manager and NT networking, including security (ACEs, ACLs, and all that stuff), SQL and SQL Server, Jet and Access, JavaScript, XML, and a few other cheerful facts about the square of the hypotenuse. When I can't get the StrConv function in VB to do what I want, I bang out an COM control so I can drop into C++ with ATL and call the MLang functions without dropping a beat. It took me years to get to this point.

There are lots of other programming worlds. There's the world of people developing for BEA Weblogic who know J2EE, Oracle, and all kinds of Java things that I don't even know enough about to enumerate. There are hard core Macintosh developers who know CodeWarrior, MPW, Toolbox programming in System 6 through X, Cocoa, Carbon, and even nice obsolete things like OpenDoc that don't help any more.

Very few people, though, know more than one or two worlds, because there's just so much to learn that unless you have to work in one of these worlds for more than a couple of years, you don't really grok it all.

But learn you must.

People get kind of miffed when they go on job interviews and get rejected because, for example, they don't have Win32 (or J2EE, or Mac programming, or whatever) experience. Or they get annoyed because idiot recruiters, who would not know an MSMQ if it bit them in the tailbone, call them up and ask if they "have 5 years MSMQ."

Until you've done Windows programming for a while, you may think that Win32 is just a library, like any other library, you'll read the book and learn it and call it when you need to. You might think that basic programming, say, your expert C++ skills, are the 90% and all the APIs are the 10% fluff you can catch up on in a few weeks. To these people I humbly suggest: times have changed. The ratio has reversed.

Very few people get to work on low level C algorithms that just move bytes around any more. Most of us spend all our time these days calling APIs, not moving bytes. Someone who is a fantastic C++ coder with no API experience only knows about 10% of what you use every day writing code that runs on an API. When the economy is doing well, this doesn't matter. You still get jobs, and employers pay the cost of your getting up to speed on the platform. But when the economy is a mess and 600 people apply for every job opening, employers have the luxury of choosing programmers who are already experts at the platform in question. Like programmers who can name four ways to FTP a file from Visual Basic code and the pros and cons of each.

The huge surface area of all these worlds of programming leads to pointless flame wars over whose world is better. Here's a smug comment somebody anonymously made on my discussion board:

"Just one more reason why I'm glad to be living in the 'free world.' Free as in speech (almost) and freedom from pandering to things like setup programs and the registry - just to name a few."

I think this person was trying to say that in the Linux world they don't write setup programs. Well, I hate to disappoint you, but you have something just as complicated: imake, make, config files, and all that stuff, and when you're done, you still distribute applications with a 20KB INSTALL file full of witty instructions like "You're going to need zlib" (what's that?) or "This may take a while. Go get some runts." (Runts are a kind of candy, I think.) And the registry -- instead of one big organized hive of name/value pairs, you have a thousand different file formats, one per application, with .whateverrc and foo.conf files living all over the place. And emacs wants you to learn how to program lisp if you're going to change settings, and each shell wants you to learn its personal dialect of shell script programming if you want to change settings, and on and on.

People who only know one world get really smarmy, and every time they hear about the complications in the other world, it makes them think that their world doesn't have complications. But they do. You've just moved beyond them because you are proficient in them. These worlds are just too big and complicated to compare any more. Lord Palmerston: "The Schleswig-Holstein question is so complicated, only three men in Europe have ever understood it. One was Prince Albert, who is dead. The second was a German professor who became mad. I am the third and I have forgotten all about it." The software worlds are so huge and complicated and multifaceted that when I see otherwise intelligent people writing blog entries saying something vacuous like "Microsoft is bad at operating systems," frankly, they just look dumb. Imagine trying to summarize millions of lines of code with hundreds of major feature areas created by thousands of programmers over a decade or two, where no one person can begin to understand even a large portion of it. I'm not even defending Microsoft, I'm just saying that big handwavy generalizations made from a position of deep ignorance is one of the biggest wastes of time on the net today.

Frequent readers, by now, have noticed that I've been thinking of the problem of how one might deliver an application on Linux, Macintosh, and Windows without paying disproportionately for the Linux and Macintosh versions. For this you need some kind of cross-platform library.

Java attempted this but Sun didn't grok GUIs well enough to deliver really slick native-feeling applications. Like the space alien in Star Trek watching Earth through a telescope, they knew exactly what human food was supposed to look like but they didn't realize it was supposed to taste like something. Java apps have menus in the right places but there are all these keyboard things that don't work the same way as every other Windows app and their tabbed dialogs look a little scary. And there is no way, no matter how hard you try, to make their menubars look exactly like Excel's menubars. Why? Because Java doesn't give you a very good way to drop down to the native facilities whenever the abstraction fails. When you're programming in AWT, you can't figure out the HWND of a window, you can't call the Microsoft APIs, and you certainly can't intercept WM_PAINT and do it differently. And Sun made it plenty clear that if you tried to do that, you weren't Pure. You were Polluted, and to hell with you.

After a number of highly publicized failures to build GUIs with Java (e.g. Corel's Java Office suite and Netscape's Javagator), enough people know to stay away from this world. Eclipse built their own windowing library from the ground up using native widgets just so they could write Java code that had a reasonably native look and feel.

The Mozilla engineers decided to address the cross platform problem with their own invention called XUL. So far, I'm impressed. Mozilla finally got to the point where it tastes like real food. Even my favorite bugaboo, Alt+Space N to minimize a window, works in Mozilla; it took them long enough but they did it.

Mitch Kapor, who founded Lotus and created 123, decided for his next application to go with something called wxWindows and wxPython for cross platform support.

Which is better, XUL, Eclipse's SWT, or wxWindows? I don't know. They are all such huge worlds that I couldn't really evaluate them and tell. It's not enough to read the tutorials. You have to sweat and bleed with the thing for a year or two before you really know it's good enough or realize that no matter how hard you try you can't make your UI taste like real food. Unfortunately, for most projects, you have to decide on which world to use before you can write the first line of code, which is precisely the moment when you have the least information. At a previous job we had to live with some pretty bad architecture because the first programmers used the project to teach themselves C++ and Windows programming at the same time. Some of the oldest code was written without any comprehension of event-driven programming. The core string class (of course, we had our own string class) was a textbook example of all the mistakes you could make in designing a C++ class. Eventually we cleaned up and refactored a lot of that old code but it haunted us for a while.

So for now, my advice is this: don't start a new project without at least one architect with several years of solid experience in the language, classes, APIs, and platforms you're building on. If you have a choice of platforms, use the one your team has the most skills with, even if it's not the trendiest or nominally the most productive. And when you're designing abstractions or programming tools, go the extra mile to make them leak proof.


Making Wrong Code Look Wrong

Wednesday, May 11, 2005

Way back in September 1983, I started my first real job, working at Oranim, a big bread factory in Israel that made something like 100,000 loaves of bread every night in six giant ovens the size of aircraft carriers.

The first time I walked into the bakery I couldn?euro;™t believe what a mess it was. The sides of the ovens were yellowing, machines were rusting, there was grease everywhere.

?euro;œIs it always this messy??euro;? I asked.

?euro;œWhat? What are you talking about??euro;? the manager said. ?euro;œWe just finished cleaning. This is the cleanest it?euro;™s been in weeks.?euro;?

Oh boy.

It took me a couple of months of cleaning the bakery every morning before I realized what they meant. In the bakery, clean meant no dough on the machines. Clean meant no fermenting dough in the trash. Clean meant no dough on the floors.

Clean did not mean the paint on the ovens was nice and white. Painting the ovens was something you did every decade, not every day. Clean did not mean no grease. In fact there were a lot of machines that needed to be greased or oiled regularly and a thin layer of clean oil was usually a sign of a machine that had just been cleaned.

The whole concept of clean in the bakery was something you had to learn. To an outsider, it was impossible to walk in and judge whether the place was clean or not. An outsider would never think of looking at the inside surfaces of the dough rounder (a machine that rolls square blocks of dough into balls, shown in the picture at right) to see if they had been scraped clean. An outsider would obsess over the fact that the old oven had discolored panels, because those panels were

After two months in the bakery, you learned how to ?euro;œsee?euro;? clean.

Code is the same way.

When you start out as a beginning programmer or you try to read code in a new language it all looks equally inscrutable. Until you understand the programming language itself you can?euro;™t even see obvious syntactic errors.

During the first phase of learning, you start to recognize the things that we usually refer to as ?euro;œcoding style.?euro;? So you start to notice code that doesn?euro;™t conform to indentation standards and Oddly-Capitalized variables.

It?euro;™s at this point you typically say, ?euro;œBlistering Barnacles, we?euro;™ve got to get some consistent coding conventions around here!?euro;? and you spend the next day writing up coding conventions for your team and the next six days arguing about the One True Brace Style and the next three weeks rewriting old code to conform to the One True Brace Style until a manager catches you and screams at you for wasting time on something that can never make money, and you decide that it?euro;™s not really a bad thing to only reformat code when you revisit it, so you have about half of a True Brace Style and pretty soon you forget all about that and then you can start obsessing about something else irrelevant to making money like replacing one kind of string class with another kind of string class.

As you get more proficient at writing code in a particular environment, you start to learn to see other things. Things that may be perfectly legal and perfectly OK according to the coding convention, but which make you worry.

For example, in C:

char* dest, src;

This is legal code; it may conform to your coding convention, and it may even be what was intended, but when you?euro;™ve had enough experience writing C code, you?euro;™ll notice that this declares dest as a char pointer while declaring src as merely a char, and even if this might be what you wanted, it probably isn?euro;™t. That code smells a little bit dirty.

Even more subtle:

if (i != 0)
    foo(i);

In this case the code is 100% correct; it conforms to most coding conventions and there?euro;™s nothing wrong with it, but the fact that the single-statement body of the ifstatement is not enclosed in braces may be bugging you, because you might be thinking in the back of your head, gosh, somebody might insert another line of code there

if (i != 0)
    bar(i);
    foo(i);

?euro;?and forget to add the braces, and thus accidentally make foo(i)unconditional! So when you see blocks of code that aren?euro;™t in braces, you might sense just a tiny, wee, soupçon of uncleanliness which makes you uneasy.

OK, so far I?euro;™ve mentioned three levels of achievement as a programmer:

1. You don?euro;™t know clean from unclean.

2. You have a superficial idea of cleanliness, mostly at the level of conformance to coding conventions.

3. You start to smell subtle hints of uncleanliness beneath the surface and they bug you enough to reach out and fix the code.

There?euro;™s an even higher level, though, which is what I really want to talk about:

4. You deliberately architect your code in such a way that your nose for uncleanliness makes your code more likely to be correct.

This is the real art: making robust code by literally inventing conventions that make errors stand out on the screen.

So now I?euro;™ll walk you through a little example, and then I?euro;™ll show you a general rule you can use for inventing these code-robustness conventions, and in the end it will lead to a defense of a certain type of Hungarian Notation, probably not the type that makes people carsick, though, and a criticism of exceptions in certain circumstances, though probably not the kind of circumstances you find yourself in most of the time.

But if you?euro;™re so convinced that Hungarian Notation is a Bad Thing and that exceptions are the best invention since the chocolate milkshake and you don?euro;™t even want to hear any other opinions, well, head on over to Rory?euro;™s and read the excellent comix instead; you probably won?euro;™t be missing much here anyway; in fact in a minute I?euro;™m going to have actual code samples which are likely to put you to sleep even before they get a chance to make you angry. Yep. I think the plan will be to lull you almost completely to sleep and then to sneak the Hungarian=good, Exceptions=bad thing on you when you?euro;™re sleepy and not really putting up much of a fight.

An Example

Right. On with the example. Let?euro;™s pretend that you?euro;™re building some kind of a web-based application, since those seem to be all the rage with the kids these days.

Now, there?euro;™s a security vulnerability called the Cross Site Scripting Vulnerability, a.k.a. XSS. I won?euro;™t go into the details here: all you have to know is that when you build a web application you have to be careful never to repeat back any strings that the user types into forms.

So for example if you have a web page that says ?euro;œWhat is your name??euro;? with an edit box and then submitting that page takes you to another page that says, Hello, Elmer! (assuming the user?euro;™s name is Elmer), well, that?euro;™s a security vulnerability, because the user could type in all kinds of weird HTML and JavaScript instead of ?euro;œElmer?euro;? and their weird JavaScript could do narsty things, and now those narsty things appear to come from you, so for example they can read cookies that you put there and forward them on to Dr. Evil?euro;™s evil site.

Let?euro;™s put it in pseudocode. Imagine that

s = Request("name")

reads input (a POST argument) from the HTML form. If you ever write this code:

Write "Hello, " & Request("name")

your site is already vulnerable to XSS attacks. That?euro;™s all it takes.

Instead you have to encode it before you copy it back into the HTML. Encoding it means replacing " with &quot;, replacing > with &gt;, and so forth. So

Write "Hello, " & Encode(Request("name"))

is perfectly safe.

All strings that originate from the user are unsafe. Any unsafe string must not be output without encoding it.

Let?euro;™s try to come up with a coding convention that will ensure that if you ever make this mistake, the code will just look wrong. If wrong code, at least, looks wrong, then it has a fighting chance of getting caught by someone working on that code or reviewing that code.

Possible Solution #1

One solution is to encode all strings right away, the minute they come in from the user:

s = Encode(Request("name"))

So our convention says this: if you ever see Request that is not surrounded by Encode, the code must be wrong.

You start to train your eyes to look for naked Requests, because they violate the convention.

That works, in the sense that if you follow this convention you?euro;™ll never have a XSS bug, but that?euro;™s not necessarily the best architecture. For example maybe you want to store these user strings in a database somewhere, and it doesn?euro;™t make sense to have them stored HTML-encoded in the database, because they might have to go somewhere that is not an HTML page, like to a credit card processing application that will get confused if they are HTML-encoded. Most web applications are developed under the principle that all strings internally are not encoded until the very last moment before they are sent to an HTML page, and that?euro;™s probably the right architecture.

We really need to be able to keep things around in unsafe format for a while.

OK. I?euro;™ll try again.

Possible Solution #2

What if we made a coding convention that said that when you write out any string you have to encode it?

s = Request("name")

// much later:
Write Encode(s)

Now whenever you see a naked Write without the Encode you know something is amiss.

Well, that doesn?euro;™t quite work?euro;?sometimes you have little bits of HTML around in your code and you can?euro;™t encode them:

If mode = "linebreak" Then prefix = "<br>"

// much later:
Write prefix

This looks wrong according to our convention, which requires us to encode strings on the way out:

Write Encode(prefix)

But now the "<br>", which is supposed to start a new line, gets encoded to &lt;br&gt; and appears to the user as a literal < b r >. That?euro;™s not right either.

So, sometimes you can?euro;™t encode a string when you read it in, and sometimes you can?euro;™t encode it when you write it out, so neither of these proposals works. And without a convention, we?euro;™re still running the risk that you do this:

s = Request("name")

...pages later...
name = s

...pages later...
recordset("name") = name // store name in db in a column "name"

...days later...
theName = recordset("name")

...pages or even months later...
Write theName

Did we remember to encode the string? There?euro;™s no single place where you can look to see the bug. There?euro;™s no place to sniff. If you have a lot of code like this, it takes a ton of detective work to trace the origin of every string that is ever written out to make sure it has been encoded.

The Real Solution

So let me suggest a coding convention that works. We?euro;™ll have just one rule:

All strings that come from the user must be stored in variables (or database columns) with a name starting with the prefix "us" (for Unsafe String). All strings that have been HTML encoded or which came from a known-safe location must be stored in variables with a name starting with the prefix "s" (for Safe string).

Let me rewrite that same code, changing nothing but the variable names to match our new convention.

us = Request("name")

...pages later...
usName = us

...pages later...
recordset("usName") = usName

...days later...
sName = Encode(recordset("usName"))

...pages or even months later...
Write sName

The thing I want you to notice about the new convention is that now, if you make a mistake with an unsafe string, you can always see it on some single line of code, as long as the coding convention is adhered to:

s = Request("name")

is a priori wrong, because you see the result of Request being assigned to a variable whose name begins with s, which is against the rules. The result of Request is always unsafe so it must always be assigned to a variable whose name begins with ?euro;œus?euro;?.

us = Request("name")

is always OK.

usName = us

is always OK.

sName = us

is certainly wrong.

sName = Encode(us)

is certainly correct.

Write usName

is certainly wrong.

Write sName

is OK, as is

Write Encode(usName)

Every line of code can be inspected by itself, and if every line of code is correct, the entire body of code is correct.

Eventually, with this coding convention, your eyes learn to see the Write usXXX and know that it?euro;™s wrong, and you instantly know how to fix it, too. I know, it?euro;™s a little bit hard to see the wrong code at first, but do this for three weeks, and your eyes will adapt, just like the bakery workers who learned to look at a giant bread factory and instantly say, ?euro;œjay-zuss, nobody cleaned insahd rounduh fo-ah! What the hayl kine a opparashun y?euro;™awls runnin' heey-uh??euro;?

In fact we can extend the rule a bit, and rename (or wrap) the Request and Encodefunctions to be UsRequest and SEncode... in other words, functions that return an unsafe string or a safe string will start with Us and S, just like variables. Now look at the code:

us = UsRequest("name")
usName = us
recordset("usName") = usName
sName = SEncode(recordset("usName"))
Write sName

See what I did? Now you can look to see that both sides of the equal sign start with the same prefix to see mistakes.

us = UsRequest("name") // ok, both sides start with US
s = UsRequest("name") // bug
usName = us // ok
sName = us // certainly wrong.
sName = SEncode(us) // certainly correct.

Heck, I can take it one step further, by naming Write to WriteS and renaming SEncode to SFromUs:

us = UsRequest("name")
usName = us
recordset("usName") = usName
sName = SFromUs(recordset("usName"))
WriteS sName

This makes mistakes even more visible. Your eyes will learn to ?euro;œsee?euro;? smelly code, and this will help you find obscure security bugs just through the normal process of writing code and reading code.

Making wrong code look wrong is nice, but it?euro;™s not necessarily the best possible solution to every security problem. It doesn?euro;™t catch every possible bug or mistake, because you might not look at every line of code. But it?euro;™s sure a heck of a lot better than nothing, and I?euro;™d much rather have a coding convention where wrong code, at least, looked wrong. You instantly gain the incremental benefit that every time a programmer?euro;™s eyes pass over a line of code, that particular bug is checked for and prevented.

A General Rule

This business of making wrong code look wrong depends on getting the right things close together in one place on the screen. When I?euro;™m looking at a string, in order to get the code right, I need to know, everywhere I see that string, whether it?euro;™s safe or unsafe. I don?euro;™t want that information to be in another file or on another page that I would have to scroll to. I have to be able to see it right there and that means a variable naming convention.

There are a lot of other examples where you can improve code by moving things next to each other. Most coding conventions include rules like:

  • Keep functions short.
  • Declare your variables as close as possible to the place where you will use them.
  • Don?euro;™t use macros to create your own personal programming language.
  • Don?euro;™t use goto.
  • Don?euro;™t put closing braces more than one screen away from the matching opening brace.

What all these rules have in common is that they are trying to get the relevant information about what a line of code really does physically as close together as possible. This improves the chances that your eyeballs will be able to figure out everything that?euro;™s going on.

In general, I have to admit that I?euro;™m a little bit scared of language features that hide things. When you see the code

i = j * 5;

?euro;?in C you know, at least, that j is being multiplied by five and the results stored in i.

But if you see that same snippet of code in C++, you don?euro;™t know anything. Nothing. The only way to know what?euro;™s really happening in C++ is to find out what types i and j are, something which might be declared somewhere altogether else. That?euro;™s because j might be of a type that has operator* overloaded and it does something terribly witty when you try to multiply it. And i might be of a type that has operator= overloaded, and the types might not be compatible so an automatic type coercion function might end up being called. And the only way to find out is not only to check the type of the variables, but to find the code that implements that type, and God help you if there?euro;™s inheritance somewhere, because now you have to traipse all the way up the class hierarchy all by yourself trying to find where that code really is, and if there?euro;™s polymorphism somewhere, you?euro;™re really in trouble because it?euro;™s not enough to know what type i and j are declared, you have to know what type they are right now, which might involve inspecting an arbitrary amount of code and you can never really be sure if you?euro;™ve looked everywhere thanks to the halting problem (phew!).

When you see i=j*5 in C++ you are really on your own, bubby, and that, in my mind, reduces the ability to detect possible problems just by looking at code.

None of this was supposed to matter, of course. When you do clever-schoolboy things like override operator*, this is meant to be to help you provide a nice waterproof abstraction. Golly, j is a Unicode String type, and multiplying a Unicode String by an integer is obviously a good abstraction for converting Traditional Chinese to Standard Chinese, right?

The trouble is, of course, that waterproof abstractions aren?euro;™t. I?euro;™ve already talked about this extensively in The Law of Leaky Abstractions so I won?euro;™t repeat myself here.

Scott Meyers has made a whole career out of showing you all the ways they fail and bite you, in C++ at least. (By the way, the third edition of Scott?euro;™s book Effective C++ just came out; it?euro;™s completely rewritten; get your copy today!)

Okay.

I?euro;™m losing track. I better summarize The Story Until Now:

Look for coding conventions that make wrong code look wrong. Getting the right information collocated all together in the same place on screen in your code lets you see certain types of problems and fix them right away.

I?euro;™m Hungary

So now we get back to the infamous Hungarian notation.

Hungarian notation was invented by Microsoft programmer Charles Simonyi. One of the major projects Simonyi worked on at Microsoft was Word; in fact he led the project to create the world?euro;™s first WYSIWYG word processor, something called Bravo at Xerox Parc.

In WYSIWYG word processing, you have scrollable windows, so every coordinate has to be interpreted as either relative to the window or relative to the page, and that makes a big difference, and keeping them straight is pretty important.

Which, I surmise, is one of the many good reasons Simonyi started using something that came to be called Hungarian notation. It looked like Hungarian, and Simonyi was from Hungary, thus the name. In Simonyi?euro;™s version of Hungarian notation, every variable was prefixed with a lower case tag that indicated the kind of thing that the variable contained.

I?euro;™m using the word kind on purpose, there, because Simonyi mistakenly used the word type in his paper, and generations of programmers misunderstood what he meant.

If you read Simonyi?euro;™s paper closely, what he was getting at was the same kind of naming convention as I used in my example above where we decided that us meant ?euro;œunsafe string?euro;? and s meant ?euro;œsafe string.?euro;? They?euro;™re both of type string. The compiler won?euro;™t help you if you assign one to the other and Intellisense won?euro;™t tell you bupkis. But they are semantically different; they need to be interpreted differently and treated differently and some kind of conversion function will need to be called if you assign one to the other or you will have a runtime bug. If you?euro;™re lucky.

Simonyi?euro;™s original concept for Hungarian notation was called, inside Microsoft, Apps Hungarian, because it was used in the Applications Division, to wit, Word and Excel. In Excel?euro;™s source code you see a lot of rw and col and when you see those you know that they refer to rows and columns. Yep, they?euro;™re both integers, but it never makes sense to assign between them. In Word, I'm told, you see a lot of xl and xw, where xl means ?euro;œhorizontal coordinates relative to the layout?euro;? and xw means ?euro;œhorizontal coordinates relative to the window.?euro;? Both ints. Not interchangeable. In both apps you see a lot of cb meaning ?euro;œcount of bytes.?euro;? Yep, it?euro;™s an int again, but you know so much more about it just by looking at the variable name. It?euro;™s a count of bytes: a buffer size. And if you see xl = cb, well, blow the Bad Code Whistle, that is obviously wrong code, because even though xl and cb are both integers, it?euro;™s completely crazy to set a horizontal offset in pixels to a count of bytes.

In Apps Hungarian prefixes are used for functions, as well as variables. So, to tell you the truth, I?euro;™ve never seen the Word source code, but I?euro;™ll bet you dollars to donuts there?euro;™s a function called YlFromYw which converts from vertical window coordinates to vertical layout coordinates. Apps Hungarian requires the notation TypeFromType instead of the more traditional TypeToType so that every function name could begin with the type of thing that it was returning, just like I did earlier in the example when I renamed Encode SFromUs. In fact in proper Apps Hungarian the Encode function would have to be named SFromUs. Apps Hungarian wouldn?euro;™t really give you a choice in how to name this function. That?euro;™s a good thing, because it?euro;™s one less thing you need to remember, and you don?euro;™t have to wonder what kind of encoding is being referred to by the word Encode: you have something much more precise.

Apps Hungarian was extremely valuable, especially in the days of C programming where the compiler didn?euro;™t provide a very useful type system.

But then something kind of wrong happened.

The dark side took over Hungarian Notation.

Nobody seems to know why or how, but it appears that the documentation writers on the Windows team inadvertently invented what came to be known as Systems Hungarian.

Somebody, somewhere, read Simonyi?euro;™s paper, where he used the word ?euro;œtype,?euro;? and thought he meant type, like class, like in a type system, like the type checking that the compiler does. He did not. He explained very carefully exactly what he meant by the word ?euro;œtype,?euro;? but it didn?euro;™t help. The damage was done.

Apps Hungarian had very useful, meaningful prefixes like ?euro;œix?euro;? to mean an index into an array, ?euro;œc?euro;? to mean a count, ?euro;œd?euro;? to mean the difference between two numbers (for example ?euro;œdx?euro;? meant ?euro;œwidth?euro;?), and so forth.

Systems Hungarian had far less useful prefixes like ?euro;œl?euro;? for long and ?euro;œul?euro;? for ?euro;œunsigned long?euro;? and ?euro;œdw?euro;? for double word, which is, actually, uh, an unsigned long. In Systems Hungarian, the only thing that the prefix told you was the actual data type of the variable.

This was a subtle but complete misunderstanding of Simonyi?euro;™s intention and practice, and it just goes to show you that if you write convoluted, dense academic prose nobody will understand it and your ideas will be misinterpreted and then the misinterpreted ideas will be ridiculed even when they weren?euro;™t your ideas. So in Systems Hungarian you got a lot of dwFoo meaning ?euro;œdouble word foo,?euro;? and doggone it, the fact that a variable is a double word tells you darn near nothing useful at all. So it?euro;™s no wonder people rebelled against Systems Hungarian.

Systems Hungarian was promulgated far and wide; it is the standard throughout the Windows programming documentation; it was spread extensively by books like Charles Petzold?euro;™s Programming Windows, the bible for learning Windows programming, and it rapidly became the dominant form of Hungarian, even inside Microsoft, where very few programmers outside the Word and Excel teams understood just what a mistake they had made.

And then came The Great Rebellion. Eventually, programmers who never understood Hungarian in the first place noticed that the misunderstood subset they were using was Pretty Dang Annoying and Well-Nigh Useless, and they revolted against it. Now, there are still some nice qualities in Systems Hungarian, which help you see bugs. At the very least, if you use Systems Hungarian, you?euro;™ll know the type of a variable at the spot where you?euro;™re using it. But it?euro;™s not nearly as valuable as Apps Hungarian.

The Great Rebellion hit its peak with the first release of .NET. Microsoft finally started telling people, ?euro;œHungarian Notation Is Not Recommended.?euro;? There was much rejoicing. I don?euro;™t even think they bothered saying why. They just went through the naming guidelines section of the document and wrote, ?euro;œDo Not Use Hungarian Notation?euro;? in every entry. Hungarian Notation was so doggone unpopular by this point that nobody really complained, and everybody in the world outside of Excel and Word were relieved at no longer having to use an awkward naming convention that, they thought, was unnecessary in the days of strong type checking and Intellisense.

But there?euro;™s still a tremendous amount of value to Apps Hungarian, in that it increases collocation in code, which makes the code easier to read, write, debug, and maintain, and, most importantly, it makes wrong code look wrong.

Before we go, there?euro;™s one more thing I promised to do, which is to bash exceptions one more time. The last time I did that I got in a lot of trouble. In an off-the-cuff remark on the Joel on Software homepage, I wrote that I don?euro;™t like exceptions because they are, effectively, an invisible goto, which, I reasoned, is even worse than a goto you can see. Of course millions of people jumped down my throat. The only person in the world who leapt to my defense was, of course, Raymond Chen, who is, by the way, the best programmer in the world, so that has to say something, right?

Here?euro;™s the thing with exceptions, in the context of this article. Your eyes learn to see wrong things, as long as there is something to see, and this prevents bugs. In order to make code really, really robust, when you code-review it, you need to have coding conventions that allow collocation. In other words, the more information about what code is doing is located right in front of your eyes, the better a job you?euro;™ll do at finding the mistakes. When you have code that says

dosomething();
cleanup();

?euro;?your eyes tell you, what?euro;™s wrong with that? We always clean up! But the possibility that dosomething might throw an exception means that cleanupmight not get called. And that?euro;™s easily fixable, using finally or whatnot, but that?euro;™s not my point: my point is that the only way to know that cleanup is definitely called is to investigate the entire call tree of dosomething to see if there?euro;™s anything in there, anywhere, which can throw an exception, and that?euro;™s ok, and there are things like checked exceptions to make it less painful, but the real point is that exceptions eliminate collocation. You have to look somewhere else to answer a question of whether code is doing the right thing, so you?euro;™re not able to take advantage of your eye?euro;™s built-in ability to learn to see wrong code, because there?euro;™s nothing to see.

Now, when I?euro;™m writing a dinky script to gather up a bunch of data and print it once a day, heck yeah, exceptions are great. I like nothing more than to ignore all possible wrong things that can happen and just wrap up the whole damn program in a big ol?euro;™ try/catch that emails me if anything ever goes wrong. Exceptions are fine for quick-and-dirty code, for scripts, and for code that is neither mission critical nor life-sustaining. But if you?euro;™re writing an operating system, or a nuclear power plant, or the software to control a high speed circular saw used in open heart surgery, exceptions are extremely dangerous.

I know people will assume that I?euro;™m a lame programmer for failing to understand exceptions properly and failing to understand all the ways they can improve my life if only I was willing to let exceptions into my heart, but, too bad. The way to write really reliable code is to try to use simple tools that take into account typical human frailty, not complex tools with hidden side effects and leaky abstractions that assume an infallible programmer.

More Reading

If you're still all gung-ho about exceptions, read Raymond Chen's essay Cleaner, more elegant, and harder to recognize. "It is extraordinarily difficult to see the difference between bad exception-based code and not-bad exception-based code... exceptions are too hard and I'm not smart enough to handle them."

Raymond's rant about Death by Macros, A rant against flow control macros, is about another case where failing to get information all in the same place makes code unmaintainable. "When you see code that uses [macros], you have to go dig through header files to figure out what they do."

For background on the history of Hungarian notation, start with Simonyi's original paper, Hungarian Notation. Doug Klunder introduced this to the Excel team in a somewhat clearer paper. For more stories about Hungarian and how it got ruined by documentation writers, read Larry Osterman's post, especially Scott Ludwig's comment, or Rick Schaut's post.


Michael E. Porter's Competitive Strategy

Thursday, June 21, 2001

As research for the next book I'm supposed to be writing, I've been reading Michael E. Porter's book Competitive Strategy. It's a brilliant book and it is considered the standard text on the subject, read by freshly hatched management consultants worldwide. But it's drier than school cafeteria chicken, and every time I read two pages, I have to take a nap.

It's worth it, so I've been taking a lot of naps lately. I'm only on page 57.

One thing the book emphasizes is that you should analyze your competition and try to guess how they will react to your move. Here's one question to ask yourself:

Are there any regulatory, antitrust, or other governmental or social constraints on the behavior of the firm that will affect such things as its reaction to moves of a smaller competitor or the probability that it will try to gain a larger market share? Has the competitor had any antitrust problems in the past? ... Has it entered into any consent decrees? Such restraints or even just a history may sensitize a firm so that it foregoes reacting to strategic events unless some essential element [of] its business is threatened. The firm attempting to capture a small share of a market from an industry leader can enjoy some protection as a result of such constraints, for example. [page 53]

Of course, antitrust reminds me of Microsoft. Microsoft has always acted, against all credibility, as if it didn't do anything remotely wrong -- pundits ask "how can Microsoft executives be the only ones in the world who are so delusional about the legality their anticompetitive behavior?" Microsoft's strategy has been to proclaim their innocence repeatedly; to continue to compete aggressively in all areas, and to merrily do new things, such as requiring Passport accounts for everything, which are bound to get them into anti-trust trouble yet again. Why?

If you've been working in the software industry, you've probably seen zillions of good new product ideas quashed by some executive who asks, "what if Microsoft enters this space and makes all the profits go away?" Lemming VCs generally have a policy of automatically saying "no" to any investment that could be seen as competing with Microsoft or which might prod Microsoft to create a new product in response. This herd fear of Microsoft in the software industry is enormously beneficial to Bill ... lots of potential competition never gets off the ground because of fear that Microsoft will squash it.

This benefit would be lost if Microsoft acted even slightly concerned about the Justice Department. Put 2 and 2 together: if Microsoft were to act contrite and concede even the slightest point in the Justice Department's anti-trust case, hundreds of independent software companies and VCs would see this as their opportunity to enter Microsoft's markets and try to get a small piece of market share, knowing that Microsoft would be sensitized against appearing to be monopolistic and therefore might allow smaller competitors to survive. Bill Gates remembers working with IBM on OS/2 in the late 1980s, when it was completely impossible to get anything accomplished at IBM. Microsoft has an institutional memory of how ineffectual that company was against competitors: everything IBM did had to be reviewed by internal consent decree committees to make sure it wasn't monopolistic, and it was no problem for young companies (like, um, Microsoft) to take big bites out of IBM's ankles without provoking a reaction.

Seen in that context, Microsoft's squashing of Netscape could well have been intended as a stern warning across the bow of Kleiner Perkins to stay the hell out of their garden; as much as Marc Andreessen's childish threats to replace Windows was threatening, it was even more threatening that John Doerr seemed to have lost the fear of Microsoft that is vital to Microsoft's dominance of the software industry.

It only makes sense to read Porter after you've spent a few years reading the business press; every sentence will remind you of dozens of examples you've seen across other industries and the whole thing will make sense. That's the other reason this book takes me so long to read (besides the obligatory naps caused by it's soporific nature): every sentence makes me look up at the ceiling, say "Ah-ha!" and rethink some aspect of how the software industry works in a new light.


Microsoft Goes Bonkers

Saturday, July 22, 2000

Microsoft's latest announcement, called Microsoft .NET, while touted by the likes of Fortune Magazine as a huge "revolution", is really nothing but vaporware, and I think it proves that something has gone very, very wrong in Redmond.

With vaporware, you promise all kinds of features and products that you simply can't sell because you don't really have them. But .NET is worse than vaporware. In their blasé loftiness, Microsoft isn't even bothering to provide the vapor itself.

Read the white paper closely, and you'll see that for all the hoopla, .NET is just a thin cloud of FUD. There's no there there. Try as you might to grasp onto something, the entire white paper does not say anything. The harder you grasp, the more it slips right through your fingers.

I'm not saying that there's nothing new in .NET. I'm saying that there's nothing there at all.

Look at some of this:

Everyone believes the Web will evolve, but for that evolution to be truly empowering for developers, businesses and consumers, a radical new vision is needed. Microsoft's goal is to provide that vision and the technology to make it a reality. [from "Microsoft .NET: Realizing the Next Generation", June 2000].

How about this:

The Microsoft .NET vision means empowerment for consumers, businesses, software developers and the entire industry. It means unleashing the full potential of the Internet. And it means the Web the way you want it. [Ibid]

What's going on here? I couldn't find one single idea that could actually be implemented in a software product in that entire white paper. Instead of providing a list of features, Microsoft provides a list of amorphous "benefits" like this one:

Web sites become flexible services that can interact, and exchange and leverage each other's data. [Ibid]

That's a "feature" of this exciting .NET architecture. The fact that it is so broad, vague, and high level that it doesn't mean anything at all doesn't seem to be bothering anyone. Or how about:

Microsoft .NET makes it possible to find services and people with which to interact. [Ibid]

Oh, joy! Five years after Altavista went live, and two years after Larry Page and Sergei Brin actually invented a radically better search engine, Microsoft is pretending like there's no way to search on the Internet and they're going to solve this problem for us. The whole document is exactly like that.

There are two things going on here. Microsoft has some great thinkers. When great thinkers think about problems, they start to see patterns. They look at the problem of people sending each other word-processor files, and then they look at the problem of people sending each other spreadsheets, and they realize that there's a general pattern: sending files. That's one level of abstraction already. Then they go up one more level: people send files, but web browsers also "send" requests for web pages. Those are both sending operations, so our clever thinker invents a new, higher, broader abstraction called messaging, but now it's getting really vague and nobody really knows what they're talking about any more.

And if you go too far up, abstraction-wise, you run out of oxygen. Sometimes smart thinkers just don't know when to stop, and they create these absurd, all-encompassing, high-level pictures of the universe that are all good and fine, but don't actually mean anything at all.

And that seems to be what happened here.

The next generation of the Windows desktop platform, Windows.NET supports productivity, creativity, management, entertainment and much more, and is designed to put users in control of their digital lives. [Ibid]

This stuff is so abstract it's impossible to criticize. Who doesn't want an operating system that supports productivity? Great feature! Get me one of those spiffy new operating systems with the productivity feature! Problem: How exactly is Microsoft going to do it? For the last 20 years of software, productivity improvements have been gradual and incremental. Have they suddenly discovered a new chemical compound that will make their operating system more productive? I don't think they have. I think they're bluffing. FUD and vaporware.

The scary thing is, they're earnest.

I know Microsoft; worked there for three years. I know the kind of people that wrote this document. Bill Gates almost certainly had a very significant role in it; that's why he gave up the CEO position, so he could work on this stuff. I don't think that Microsoft created this document because they needed some vaporware. These are super-smart people.

I actually think that they earnestly think they're inventing the future, as well as they know how. They've looked at every Microsoft product, from Hotmail to SQL Server, and tried to fit them into a Bold New Vision Thing. But the trouble is that nobody there is actually inventing anything earthshaking. Which isn't surprising: not because Microsoft is stupid, which they're not, but because earthshaking new inventions are so rare and Microsoft only has a finite number of smart people. Only one person in the whole world invented Napster, and he didn't work for Microsoft. Microsoft desperately wants to believe that it can manufacture revolution, but even in the Cambrian explosion of the Internet, there are only a handful of truly revolutionary ideas per year, and the chances that one of them will happen inside the tiny world of Bill Gates and the knights of the Redmond table are vanishingly small. The chances are even smaller when you consider that a typical smart programmer working in the bowels of Microsoft on display drivers for Windows NT, who has a great idea, is probably not going to get his idea listened to.

The only thing concrete that you can discern from the white paper is that software should be a subscription service you get over the Internet, not something you install from a CD-ROM.

To a customer, getting your word processor via Internet subscription rather than via a CD-ROM might be a small benefit, but, um, no, not really. It doesn't really solve any customer problem. Getting bug fixes over the Internet? Great. I can already do that. I've been downloading patches for Microsoft products for 7 years, and now it's quite automatic. Getting new versions? What's the point, if the only thing that the new version does is make it easier to get new versions! They've hardly added a single new feature to Word for the last three releases, except that at some point, they did something bizarre to make it "easy" to position pictures, and I can never get pictures to go where I want them.

The truth is, Microsoft noticed way back in 1991 that an increasing amount of their revenue came from upgrades, and that it's hard to get everybody to upgrade, and they've been trying to get their customers to agree to a subscription model for buying software for almost a decade. But it hasn't worked because the customers don't want it. Microsoft sees .NET as a way to finally enforce the subscription model which suits their bottom line.

It almost seems as if Microsoft .NET doesn't fill a single customer need, it only fills Microsoft's need to find something for 10,000 programmers to do for the next 10 years. We all know it's been a long time since they've thought of a new word processing feature that anybody needs, so what else are all those programmers going to do?

The Bright Side of the "Vision Thing"

Old joke: A man goes to the psychiatrist. The shrink shows him a picture of a bird and says "What does this make you think of?" The man says: "Sex." The shrink shows him a picture of a tree. "OK, what does this make you think of?" The man says: "Sex." Picture of a train. "Sex." A house. "Sex."

"My God!" says the shrink. "You're obsessed with sex!"

"I'm obsessed with sex!?" says the man. "You're the one who keeps showing me dirty pictures!"

Ya see, the bright side of vague documents like the .NET white paper is that they are a kind of Rorschach test. People read them with preconceived ideas, and since the document is so vague, they think that Microsoft is reiterating their ideas. Dave Winer, president of UserLand software, has many interesting, innovative ideas about software. When he read about Microsoft .NET, he assumed that Microsoft was finally recognizing the same ideas that he'd been talking about for two years. Dave, you give them too much credit. They are completely clueless compared to you. They are playing the trick of psychic hotlines and newspaper horoscopes: by feeding you cloudy, meaningless generalizations, you fall into their trap of thinking that they read your mind. "Today the planetary alignment is such that you will take a big step forward to achieve your goals." The difference is that Dave has real, concrete ideas that can translate into real software, while Microsoft is still in the kind of lalaland they were in 6 years ago when they were talking about how "Cairo" would provide "Information At Your Fingertips," a vision that the Internet fulfilled and Cairo didn't.

So hopefully, all this meaningless blah blah will actually get somebody's creative juices going (as it has at UserLand) and lead to some real innovations. But these innovations are probably more likely to come from outside Microsoft than inside.

Postscript: "But Wait," you say. "I have the .NET SDK!"

Since this article originally appeared a lot of people have written in to say that they have the .NET SDK! It's not vapor! It's "real!"

Well, uh, yeah. What's in it? There's SOAP, a technology basically invented by Dave Winer based on XmlRpc, which I personally baked into the Juno signup process about two years ago. Microsoft is a little late to that ballgame. There's a programming language, C#, which is just Microsoft's way of saying that if they can't take over Java, dammit, they're going home and playing with their own toys. There are new versions of ADO, ASP, and some other stuff... all good stuff, but just incremental improvements. There's nothing revolutionary here, Fortune Magazine. If Microsoft's marketing wasn't working in overdrive, we'd have all this stuff anyway, and nobody would be pretending that it's some kind of computer Nirvana on the horizon.

This is the way Microsoft works: they have a product team for each product, and every year or two, that team ships a new version of their software. That's all. What we have here, ladies and gentlemen, is a pure marketing team that looked around at all the upcoming releases, decided they need a "theme" to make Microsoft look like Big Revolutionary Innovators, and ordering everyone to call their next thing ".NET". When you work at a place like Microsoft, there's nothing more frustrating than when the marketing people take over: read this response to my article from a Microsoft insider.


Mike Gunderloy's Coder to Developer

Wednesday, May 05, 2004

Note: This is my foreword to Mike Gunderloy's awesome new book, Coder to Developer. The book is now available from SYBEX.

You know what drives me crazy?

?euro;œEverything??euro;? you ask. Well, OK, some of you know me a bit too well by now.

But seriously, folks, what drives me crazy is that most software developers don?euro;™t realize just how little they know about software development.

Take, for example, me.

When I was a teenager, as soon as I finished reading Peter Norton?euro;™s famous guide to programming the IBM-PC in Assembler, I was convinced that I knew everything there was to know about software development in general. Heck, I was ready to start a software company to make a word processor, you see, and it was going to be really good. My imaginary software company was going to have coffee breaks with free donuts every hour. A lot of my daydreams in those days involved donuts.

When I got out of the army, I headed off to college and got a degree in Computer Science. Now I really knew everything. I knew more than everything, because I had learned a bunch of computer-scientific junk about linear algebra and NP completeness and frigging lambda calculus which was obviously useless, so I thought they must have run out of useful things to teach us and were scraping the bottom of the barrel.

Nope. At my first job I noticed how many things there are that many Computer Science departments are too snooty to actually teach you. Things like software teamwork. Practical advice about user interface design. Professional tools like source code control, bug tracking databases, debuggers and profilers. Business things. Computer Science departments in the most prestigious institutions just won?euro;™t teach you this stuff because they consider it ?euro;œvocational,?euro;? not academic; the kind of thing that high school dropouts learn at the local technical institute so they can have a career as an auto mechanic, or an air-conditioner repairman, or a (holding nose between thumb and forefinger) ?euro;œsoftware developer.?euro;?

I can sort of understand that attitude. After all, many prestigious undergraduate institutions see their goal as preparing you for life, not teaching you a career, least of all a career in a field that changes so rapidly any technologies you learn now will be obsolete in a decade.

Over the next decade I proceeded to learn an incredible amount about software development and all the things it takes to produce software. I worked at Microsoft on the Excel team, at Viacom on the web team, and at Juno on their email client. And, you know what? At every point in the learning cycle, I was completely convinced that I knew everything there was to know about software development.

?euro;œMaybe you?euro;™re just an arrogant sod??euro;? you ask, possibly using an even spicier word than ?euro;œsod.?euro;? I beg your pardon: this is my foreword; if you want to be rude write your own damn foreword, tear mine out of the book, and put yours in instead.

There?euro;™s something weird about software development, some mystical quality, that makes all kinds of people think they know how to do it. I?euro;™ve worked at dotcom-type companies full of liberal arts majors with no software experience or training who nevertheless were convinced that they knew how to manage software teams and design user interfaces. This is weird, because nobody thinks they know how to remove a burst appendix, or rebuild a car engine, unless they actually know how to do it, but for some reason there are all these people floating around who think they know everything there is to know about software development.

Anyway, the responsibility is going to fall on your shoulders. You?euro;™re probably going to have to learn how to do software development on your own. If you?euro;™re really lucky, you?euro;™ve had some experience working directly with top notch software developers who can teach you this stuff, but most people don?euro;™t have that opportunity. So I?euro;™m glad to see that Mike Gunderloy has taken upon himself to write the book you hold in your hands. Here you will find a well-written and enjoyable introduction to many of the most important things that you?euro;™re going to need to know as you move from being a person who can write code to being a person who can develop software. Do those sound like the same thing? They?euro;™re not. That?euro;™s roughly the equivalent of going from being a six year old who can crayon some simple words, backwards N?euro;™s and all, to being a successful novelist who writes books that receive rave reviews and sell millions of copies. Being a software developer means you can take a concept, build a team, set up state of the art development processes, design a software product, the right software product, and produce it. Not just any software product: a high quality software product that solves a problem and delights your users. With documentation. A web page. A setup program. Test cases. Norwegian versions. Bokmål and Nynorsk. Appetizers, dessert, and twenty seven eight-by-ten color glossy photographs with circles and arrows and a paragraph on the back of each one explaining what each one was. (Apologies to Arlo Guthrie.)

And then, one day, finally, perhaps when it?euro;™s too late, you?euro;™ll wake up and say, ?euro;œHmm. Maybe I really don?euro;™t know what it really takes to develop software.?euro;? And on that day only, and not one minute before, but on that day and from that day forward, you will have earned the right to call yourself a software developer. In the meantime, all is not lost: you still have my blessing if you want to eat donuts every hour.


More on Sabbaticals...

Saturday, March 18, 2000

I took a self-funded sabbatical in 1995, and I'm taking another one now. I think they're great.

In 1991, I graduated from college and set off on my first cross country journey, by Ryder van, to Redmond, Washington. My first job was at Microsoft. This was, I would like to point out, before everybody hated Microsoft. In those days, only college kids and UNIX weenies hated Microsoft because it made "toy" products and boring office software for suits. One of the kids in my class was offered a job to work on OS/2. "No way I'm going down with that ship," he said, and went to law school instead.

I was, to say the least, unhappy at the prospect of being out of school. I thrived on the social atmosphere of dorm life and was dreading the prospect of living in a dull apartment in a grey city where I didn't know anybody. Of course, that's the trick at Microsoft: most new hires are recent graduates from around the country who arrive in the waterlogged suburbia of Redmond without too many friends or social life. For the average geek, this formula means that you spend all your time at work. A typical day consisted of: wake up, walk to work (try not to step on any slugs), work until late at night, go home, watch TV, go to sleep, repeat.

My version was a little bit different, because I'm not a totally hopeless geek; I went to the gym in the evening instead of watching TV, and spent my weekends biking around Lake Washington to the U. District where I hung out at bookstores, libraries, and coffee houses and felt grumpy about not being in college anymore. But after a couple of years of this, I noticed that I wasn't developing much of a social life; I didn't have a boyfriend; everybody I knew was from Microsoft. Drabsville.

Needing a change, I moved to New York to work for Microsoft Consulting Services. At some point, I want to write a book-length treatment about that horrible hellmouth of incompetence. For now, suffice it to say that I didn't last for long. A quick calculation of my stock options showed that I had accumulated about $120,000 in 2 1/2 years at "the soft", and, by my calculations, I thought I could afford the risk of working at a startup.

I got a job offer from Pipeline, an early ISP in the New York area, and quit Microsoft. But talking to the founder and owner of Pipeline gave me some bad feelings, and thus began my first sabbatical.

Over the next 9 months or so, I did a couple of things. First, I learned. It was 1994, the Internet was starting to happen, and I had some catching up to do after living in the insular waters of Microsoft Before The Memo when it was assumed that MSN was going to compete with, and subsume, the Internet totally.

I also went through the exercise of thinking about my own startup - twice. Both exercises fell apart after a few weeks work because I didn't have the right partner, and I didn't know what I was doing, but I would like to compliment myself with the thoughts that the first startup could have become Yahoo! and the second startup could have become Vermeer (the company that became Microsoft Front Page.) I have specs somewhere on my hard drive for products that, if we had actually created them, really could have been huge Internet companies. But it's not the idea that matters, it's the execution -- an idea I will return to many times in this weblog.

Another idea that had been itching in the back of my mind was to take a cross-country bicycle trip. When these startup ideas fell through, I started planning to take a trip in the spring, as soon as the weather was warm enough. (This was before I knew that it rains a lot in the spring.) The trip was great, you can read my web log (from way back in 1995!) here.

Somewhere in Idaho, riding through an empty road, my mood changed; I felt totally rested and eager to get back to work. Sitting in the library at Boise State, I read all the computer industry trade rags enthusiastically, and was excited to see how much was changing and how many new things there were to learn. When I got home and checked my bank account, I was happy to discover that the $7000 dollars it took for the 10 week bike trip had been magically replaced through the mystical power of Microsoft stock; being out of work for eight or nine months had barely depleted my savings.

So that was sabbatical one. It took about 2 days to find another, interesting job, and I spent the next four years working: first at Viacom, then at Juno.

Last November, some of the really bad management over at Juno had just worn me down. I found it impossible to be excited any more. It was increasingly difficult to ignore the subtle and unsubtle ways that Juno managers at all levels were screwing up. Worse, the intense politicalness and arrogance of management there convinced me that I had almost no chance of changing things. I joined the flow of talented, frustrated people streaming for the doors.

I really like the formula of working for four years, and then taking one year off. This time I'm pretty convinced that when I go back to work I want to work in a real startup, as a founder. I've learned a lot about this over the years, and I've gradually come to realize that there is nothing really risky about starting a company these days. There are billions of not-very-smart venture dollars out there looking for somebody to spend; even startups pay good salaries (they have to); and the chance of having a "liquidity event" - IPO or selling the company -- is high enough that over the average career, say, working at 4 startups over a 10 year period, there is a fantanstic chance that you will make a big buttload of moolah.


Mouth Wide Shut

Wednesday, January 15, 2003

When Apple releases a new product, they tend to surprise the heck out of people, even the devoted Apple-watchers who have spent the last few months riffling through garbage dumpsters at One Infinite Loop.

Microsoft, on the other hand, can't stop talking about products that are mere glimmers in someone's eye. Testers outside the company were using .NET for years before it finally shipped.

So, which is right? Should you talk endlessly about your products under development, in hopes of building buzz, or should you hold off until you've got something ready to go?

Fog Creek's default policy so far has been Absolute Radio Silence. At times, I've considered changing that policy. After all, why not open up our complete development process to the world, let everybody peek in the windows and see what's going on? There's nothing to hide!

In my personal life, I have a policy lifted from Marlon Brando, playing a mob boss in The Freshman: “Every word I say, by definition, is a promise.” The best way to avoid breaking promises is not to make any, and that's as good a reason as I need not to talk about future versions of our products. There are four other reasons for this in software development.

Competition. I'm not overly paranoid about competition, but as soon as you have competition that's paying attention to what you do, if you discuss features that haven't shipped yet, you're handing them an opportunity. If you keep your mouth shut until you ship, you will be guaranteed at least half a year without competition before everyone else matches that cool new feature.

Underpromise and overdeliver. When you tell a customer or potential customer you're going to add a feature, they will inevitably imagine that the feature will do all kinds of things that it may not actually do. When you tell someone about the upcoming clam steaming feature, no matter how careful you are to delimit what it actually does, they will inevitably spin elaborate fantasies about how it cures baldness and warts and has a telepathic user interface. When you finally deliver something, they are bound to be disappointed. This can only hurt.

Flexibility. If you want to keep your promises, you can't talk about upcoming features and release dates unless you're willing to lock into them. This may eliminate flexibility that you need later when Murphy strikes. And if you don't keep your promises, you've ruined something that's very hard to get back: your reputation.

Simplicity. If your policy is Radio Silence, every employee understands it and can follow it. If your policy is in any way complicated, nobody is sure what to do and things leak.

Doesn't advance buzz and publicity help? I don't know. A little, but not as much as nonadvance publicity. I'm inclined to think that publicity that comes out when you can't actually buy the product is 90% wasted. Remember that incredibly big burst of Segway publicity about a year ago? With Jeff Bezos and Steve Jobs talking about how “IT” was going to revolutionize the entire universe? Cities would be reconfigured. OK, so, we all talked about the Segway, but nobody could buy one, so it's not clear that it was publicity well-spent. And it certainly seems like the same amount of publicity would have helped more if it appeared when every Walmart has Segways in stock.

One purported advantage of talking about your products under development is to get early feedback. But honestly, you don't need feedback from the entire world. Look at the poor Chandler guys; they started talking about their product before any design was done and immediately got buried under such a deluge of feedback just managing it all was impossible. Now everybody thinks Chandler is going to be All Things to Everybody. Quite a lot to live up to. I've found that you can get just as good feedback from a few hundred carefully selected customers. The other 800,000 people that send you suggestions might be sending good suggestions, but you already heard them, so they aren't adding value. (By the way, Chandler did exactly the right thing, since they are an open source project. They don't care if competitors use their ideas, and at this stage it's worth sifting through everybody's crazy feature requests if that's the price of attracting more volunteers to write the code.)

A lot of times customers and potential customers come to me and say, “FogBUGZ (or CityDesk) is almost perfect for my needs. But I need a salad spinner. When are you going to have a salad spinner? Will the next release have a salad spinner?” To these people I have to say, “I don't know. We might, or we might not. If you really can't live without a salad spinner, I'm afraid you'll have to go somewhere else.” OK, maybe we lose a sale or two by refusing to indulge in the vaporware habit. I'm a patient man, and Fog Creek is profitable so I don't need the cash. Next year we'll have salad spinners, and I won't lose the sale. It's better than selling you something that doesn't fit your needs and having you get pissed off at me, or getting a reputation as being unable to deliver.


NDAs and Contracts That You Should Never Sign

Tuesday, March 28, 2000

Over time, I've signed a lot of NDAs (non-disclosure agreements)... they seem to be the Silicon Valley equivalent of the Japanese business card. Some of them are short and simple, others are quite elaborate.

There is, however, one clause I've seen in a lot of NDAs that I consider quite unacceptable. It is a clause which forbids you to hire anybody who works for the company that is making you sign the NDA. Presumably, they think that while you are visiting them, you will hire away all their employees and put them out of business. Well, you know what? If your employees are so eager to leave, that's your problem, not mine. You need to keep your employees loyal by treating them well, not by creating arbitrary obstacles in their career. In some cases it's quite ridiculous; I recently signed this clause in a 3 page NDA by a startup company that consisted of exactly two founders, and no employees. So the founders, by putting this clause in the NDA, are doing nothing but guaranteeing that if their business doesn't work out, none of the people they met while working on the business will be able to give them jobs.

On a similar note, a lot of companies have the audacity to put non-compete clauses in their employment contracts. Typically, this says that you agree not to work for one of their "competitors" or even "potential competitors" (which is never very well defined) for a period, usually 1 or 2 years, after leaving the company.

This is completely outrageous. I signed such a contract at Microsoft without paying too much attention. When I left, I realized that because Microsoft has a finger in everything related to software, technically I could not work in my field AT ALL for 12 months after leaving Microsoft.

Of course, it is extremely rare for Microsoft to enforce this clause. Among other things, it is considered legally unenforceable in some states, like California, although in New York I have heard it can sometimes be enforced. If they are not going to enforce it, why is it in there?

At Juno, management tried to add such a clause to the contract. Of course, there was a bit of an uproar, and guess what? The employees won! The clause was removed from the standard contract.

You can win on this one. Highly skilled technical employees are in too much demand, and it's too easy to get a job. You don't have to accept this clause in your contract. If the employer absolutely, positively insists that you promise not to go work for a competitor when you leave your job, you can tell them: "fine. You don't want me to work for a year after I leave, that's fine, but if I'm going to be 'on the beach', I want you to keep paying me my salary for one year after I leave, until I can legally get a job that you approve of."

Another dangerous clause says that you agree not to hire, or cause to be hired, anybody from the company if you leave for a period of x months (usually 12 to 24). The idea is to prevent a group of people from leaving en-masse, and it's to protect against the standard practice of a manager leaving a company and taking his team along with him.

This is something you're really going to regret. Among other things, if you leave a company, then six months later someone else leaves and uses you as a reference for a new job, technically, you can't give them one. That's not nice.

And if somebody from the company leaves to do a cool startup, you can't go work for them! By signing this, you're basically cutting off this whole branch of possibilities in your career.

Suppose you want to leave to do your own startup. Your single best source of potential partners and employees are the people you know professionally. It is definitely not in your best interest to sign this clause.

I think that companies need to keep their employees loyal by treating them well, not by enforcing blind loyalty though a contract. I am not going to make the mistake of signing this clause again. Fortunately, with today's shortage of qualified programmers, the balance of power today is sharply in your favor when you start a new job, and I think you have a very high chance of getting a job without signing restrictive employment clauses.

I'm not a lawyer, and this is not legal advice!

Advertainment! Do you need to control a computer remotely, even when firewalls get in the way? My company's latest product, Fog Creek Copilot, is a remote control system that requires no setup, no configuration, and works even if both users are behind firewalls. It's designed to make remote tech support easy.

Enter your email address to receive a (very occasional) email whenever I write a major new article. You can unsubscribe at any time, of course.
Email:


Netscape Goes Bonkers

Monday, November 20, 2000

... and I'm very thankful, because Netscape 6.0 has been a terrific illustration of so many of the points I've made in Joel on Software over the last 6 months. Unfortunately, it's usually an illustration of what not to do.

Way back in April, I wrote that Netscape made the "single worst strategic mistake that any software company can make" by deciding to rewrite their code from scratch. Lou Montulli, one of the 5 programming superstars who did the original version of Navigator, emailed me to say, "I agree completely, it's one of the major reasons I resigned from Netscape." This one decision cost Netscape 3 years. That's three years in which the company couldn't add new features, couldn't respond to the competitive threads from Internet Explorer, and had to sit on their hands while Microsoft completely ate their lunch.

When they did, finally, release their software, it doesn't seem like they did very much testing. The Joel on Software discussion group was full of complaints of bugs and general instability. This is one thing you can't blame AOL for; when a new release of AOL ships, you don't usually hear such an upcry about bugs. In Top Five (Wrong) Reasons You Don't Have Testers, I wrote that Netscape "did an almost supernatural amount of damage to its reputation through their 'testing' methodology." "Testing" in quotes because their methodology seems to be to ship to millions of people and then ignore any bug reports that come in. The sheer volume of bugs, it seems, proves that rewriting code from scratch does not make for a better code base, it makes it worse. Old code doesn't rust, it gets better, as bugs are fixed. Lou Montulli again: "I laughed heartily as I got questions from one of my former employees about FTP code the he was rewriting. It had taken 3 years of tuning to get code that could read the 60 different types of FTP servers, those 5000 lines of code may have looked ugly, but at least they worked."

The biggest single complaint about Netscape 6.0 revolves around the way they reimplemented every UI widget from scratch. Indeed, they felt that this was the only way to get cross platform compatibility. (It's not. Microsoft Word and Excel are completely compatible between Mac and Windows versions, and they use native widgets and UI on each platform. When Mac Word 6.0 shipped, in which Microsoft had tried to make the interface a little bit more consistent with the Windows version, there was such a hue and cry from Mac loyalists that Microsoft had to ship a new version.) Many people complained about Netscape 6.0, for example, that you can't right-click in the edit boxes and get a context menu -- a feature which is available "for free" when you use the Windows native controls. Netscape 6 is just not very usable because it violates many rules of consistency. It's not going to win many friends who were using IE (something like 95% of the users hitting my company site are IE users) because some of the things that IE users do just don't work. For example, rather than reach for the mouse, I tend to go to a URL by typing Alt+D (which jumps to the address bar), typing the middle part of the URL (e.g. "nytimes" to go to "http://www.nytimes.com") and pressing Ctrl+Enter (in IE, this automatically adds "http://www" and ".com" for you). The Alt+D / Ctrl+Enter combination is burned into my fingers now, but when I try it with Netscape 6, of course, it doesn't work. Similarly, I've gotten into the habit of Shift+Clicking to open a link in a new window; Netscape opens the link in the same window. And I can't use the scrolling mouse button on my ThinkPad to scroll, something which works with every other window in every other application I've ever used. I can't even Alt+Space to get the context menu. I'm used to using Alt+Space N to minimize a window, and Netscape won't do it. This is the first time I've ever seen a Windows app where Alt+Space N doesn't work.

This is a lesson Microsoft learned years ago with Excel - which interprets Lotus 123 keystrokes perfectly - and Word, which is happy to go so far as to emulate WordPerfect's white-on-blue screen. (I talk about the importance of consistency in Chapter 5 of UI for Programmers).

Netscape's groovy "skinz" UI doesn't honor the system settings. Once you've set the preferred system color scheme and fonts in Windows, Netscape will completely ignore them. For me, that's just a nuisance. For vision-impaired users who need to set fonts very large or use high-contrast color schemes, it's a reason not to use Netscape.

I can't prove it, but I have to imagine that Netscape didn't have a schedule for 6.0, choosing instead to meander for years and years and suddenly "find the religion" about shipping on time when it's way too late to cut trivial features, like themes, and instead finish the important features, like standards compliance. Without a prioritized schedule, programmers will do the fun features first, not the important ones, where "fun" is very loosely defined.

Let me step back for a moment and point out one thing that I think that Netscape did right. If you've ever used Microsoft Outlook, you've probably been confused about how the Outlook Bar works:

Netscape's Sidebar has exactly the same behavior as the Outlook bar, but with a slightly different visual appearance that makes it a zillion times easier to understand:

This is so much easier to understand because it uses a real world metaphor. The visual screen conveys the program model, teaching the user how it works, thus bringing the user model in line with the program model which makes it very easy to figure out. Sadly, this nice metaphor only works in the default theme; other themes (such as Sky Pilot) seem to have been created by people who are not as sensitive to the value of metaphors.


New Server at Peer 1 Network

Monday, February 03, 2003

I've moved Joel on Software to a new server, at a colocation facility operated by Peer 1 Network. In the process of finding a new home and getting it up and running I've learned quite a bit about how web hosting works, so I thought I'd describe a bit of it here and in the process provide a glimpse Behind The Scenes.

The Fog Creek office is connected to the Internet via a single T1, which provides 1.5 Mbps of bandwidth. That's 1.5 megabits per second, the equivalent of 24 phone lines.

The T1 was provided by Savvis. The cost fluctuates depending on usage but it generally costs us something just over $1000 a month.

We had originally decided to use a T1 instead of the much cheaper alternatives (like DSL or cable modems) because T1 lines had a reputation for reliability. We ran all our servers (Fog Creek, Joel on Software, TechInterview.org) from the office because we needed a fast Internet connection to the office anyway, so why pay again for hosting? That was the theory, at least.

But the T1 turned out to be not as reliable as we thought. We had a lot of short outages, from seconds to an hour, and twice in the last year we had outages of a day or more. In both cases, the ISP did not fix them nearly as quickly as we thought they should have been able to. Although our ISP was Savvis Communications, the segment of T1 from our office to the nearest Savvis location (called the "local loop") was actually provided by Worldcom. And in fact Worldcom didn't actually provide it, they just resold a Verizon T1 under their own name. So when something went wrong with the Verizon line, we had to convince Savvis to convince Worldcom to convince Verizon to fix it. Somehow this was like pushing on a string. Outsourcing is problematic that way: when everything works perfectly, you don't notice, but when things go wrong, it's much harder to get them fixed.

You may be wondering if a single server connected to a T1 was enough for all the traffic this site gets, about 100,000 hits a day, more like 500,000 when I write a new article or get mentioned on slashdot. The short answer is, it was fine. Since I publish the site using CityDesk which simply generates a bunch of plain HTML files, the server doesn't have to do anything tricky like generate each page from a database on the fly.

Anyway, we're about to move to a new office, so a decision had to be made about whether to take the T1 with us, or do something else. The long outages convinced me we needed a real colocation facility. "Colocation" means that you pay someone with a room with extremely high speed Internet access, usually redundant, to let you put your own computer on a shelf in that room. A good colocation facility, or "colo" as they're called, also has great air conditioning, a big generator to provide power if the lights go out, and huge batteries to keep the whole thing running during the few minutes it takes to get the generator started up. It also has a lot better security than average, all kinds of exciting fire extinguishing equipment, and locked cages to keep one company's servers from being tampered with by another company that uses the same colo.

"Colocation" is not the same as "web hosting." Typically a web host will host your web site on their own servers. They administer the servers; you just put your files up and they serve them. Whereas colo means you buy and install and administer your own server hardware. We chose colocation for a number of reasons, but primarily because it gives us the flexibility to run any software we want.

If you're in the market for colocation, especially in a major metropolitan area, you probably have quite a few choices. By the way, there's no reason you have to get colocation in the same city where you're located -- you can just build a server and then ship it to the colo. The colo's engineers put it on the rack for you, plug it into power and the net, and turn it on.

A quick google search came up with a list of about eight companies that offered colocation in New York, so I called them all for price quotes. For the most part, the quote is based on how much space you need, how much bandwidth you need, and possibly how much power you need. There may be some other charges for things like IP addresses, but those are nickels 'n' dimes. Space is measured in racks. A single rack is about 6 or 7 feet high and always exactly 19 inches wide. You might rent a half rack or quarter rack if you have less equipment. If you only have one or two machines, you'll see things quoted in U's. One U is about 1.75 inches and represents the vertical distance from one set of screw holes to the next set on a rack; virtually all equipment that is intended to be mounted on a rack is a round multiple of 1U in height. The thinnest servers are 1U pizza-boxes. Our new server, a Dell 2650, is 2U in height; I chose this model because it had room for 5 hot-swappable hard drives. Hot swappable means you can plug them in and pull them out while the computer is running, so we can add some more storage without turning off the server and messing around with screwdrivers.

There are different schemes for paying for bandwidth. The most common, called capped bandwidth, is simply to pay for the bandwidth that's available. For example, you might get 1 Mbps of bandwidth capped, and you can use all of it or none of it and pay the same amount. The network administrator will program the router to cap your usage at that amount.

Another common system is where they set you up with a ton of bandwidth, but you only pay for the 95% percentile of the actual bandwidth used. That means that they sample how much bandwidth you're using every ten minutes over the course of a month. At the end of the month, they throw away the top 5% of all the samples, and charge you based on the maximum bandwidth you used during the other 95% of the time. This method means if you have a really big spurt in usage for a few minutes you won't have to pay for it. This is what's called "burstable bandwidth."

Anyway, I've been running this server for a while, so I had a pretty good idea of how much bandwidth it was going to need, and I had picked out a nice 2U server, so I took this info to the eight colocation providers I found for quotes. What I found clustered into three groups:

  • high end snobs, who told me they never do less than full racks, and I couldn't afford their services. These were usually companies from the Dot Com days that built ridiculously fancy offices and have marble bathrooms in their colos and named gigantic sports stadiums after themselves, and are now in bankrupcy court. When they would quote me a price, it was in the range of $1000 to $1500 a month.
  • reasonable small companies who had grown slowly and carefully, with slightly grimy colo facilities because they didn't waste money on trivial stuff that doesn't contribute to the bottom line like sweeping the floors or being polite to potential customers. They usually wanted $400 or $500. But luckily there was a third group, consisting of,
  • Peer 1 Network, a super friendly Canadian company that took one look at Joel on Software and said, "hey, nice site, we'll host it for free for ya." OK, I like free.

I liked just about everything I saw about Peer 1. They have just built a spiffy new facility in New York which connects to their cross-Canadian backbone. A Google search convinced me that their customers were all happy campers. And if you ever get a choice of whether to deal with Canadians or New Yorkers, I know it's unpatriotic of me to say this, but, go with the Canadians. Peer 1 is a small enough company that they care about your business, something which the giant bankrupt telcos don't. And they had a really smart network engineer working out of the New York colo who took the time to teach me a lot of the things I would know if I were a real network administrator as opposed to a software developer pretending to be a network administrator.

I got the Dell delivered straight to Peer 1's facility which is down the block from the New York Stock Exchange. When it arrived, Michael and I grabbed the Win2K CD-ROM and took the subway downtown to set it up. Most servers run "headless," that is, without a keyboard or monitor, so Peer 1 had a keyboard, monitor, and mouse on a rolling trolley available for our use so we could set up the system. (You can see the temporary wires in the picture at right.) We installed Windows 2000 and got it completely up to date with patches before connecting it to the live Internet. The server has two separate power supplies and two power cables in case one fails or falls out. The dual power cables also means that if you need to move the server or plug it in somewhere else, you can do this using extension cords without ever turning the machine off.

It had three separate ethernet jacks: you use one of them for remote administration. The remote administration jack is really quite neat. This is a little computer with its own network jack which lives inside the big computer. Even if the main computer is off or hung, I can connect to the administration computer over the Internet using a web browser and tell it to reboot or turn on the main computer. Slick!

Peer 1 only gave us a single Internet drop in our cage -- just a regular Cat 5 ethernet cable -- so I ran out to J&R Computer World and bought a $35 Linksys 5 port switch so we could hook up all three network jacks.

One last step before connecting to the Internet: we turned on packet filters to block all incoming connections except for traffic from the Fog Creek office.

Once we got the computer on the net, we tried www.google.com, which redirected immediately to www.google.ca. Cool! They think we're in Canada! Our work done, we locked up the cage and came back to the Fog Creek office. Since then, everything we need to do with this machine can be done over the net, using Windows' built-in Terminal Services. This feels exactly like sitting at the machine. With any luck, it will be years until I visit this computer again.


Nothing is as Simple as it Seems

Monday, March 04, 2002

We had a little usability problem in CityDesk.

Here was the problem: you could import files from the web using a menu command ("Import Web Page"). And you could import files from a disk into CityDesk by dragging them with the mouse. But there was no menu command to import files from a disk. So either people didn't discover that it was possible, or people tried to use the Import Web Page feature to import from disk, which didn't work right.

I thought that it would be easy to fix with a two page wizard. Roughly speaking, page one of the wizard would ask you "Where do you want to import from?" If you chose "disk," page two would prompt you for a file. If you chose "web," page two would prompt you for a URL.

I almost started implementing this, but something stopped me, and instead, I started to write a mini-spec. Here's the spec, in its entirety:

Page One
Where do you want to import from? (Disk/Web)

Page Two (Disk)
Standard File/Open dialog

Page Two (Web)
URL prompt with mini-web-browser

Suddenly something occurred to me. Can you put the Windows standard file open dialog, which is usually supplied in toto by the OS, into a wizard?

Hmm.

I investigated. Yes, you can, but it's no fun and takes a few hours of work. How could I make this not be a wizard? I rewrote the spec:

Two Menu Items:
1) Import Web Page From Internet -> Pops Up URL Dialog
2) Import Web Page From Disk -> Pops Up File Open Dialog

Much better. Three minutes of design work saved me hours of coding.

If you've spent more than 20 minutes of your life writing code, you've probably discovered a good rule of thumb by now: nothing is as simple as it seems.

Something as simple as copying a file is full of perils. What if the first argument is a directory? What if the second argument is a file? What if a file with the same name already exists in the destination? What if you don't have write permission?

What if the copy fails in the middle? What if the destination is on a remote machine which is available, but which requires authentication to continue? What if the files are large and the link is slow and you need to show a progress indicator? What if the transfer speed slows down to almost zero... when do you give up and return an error message?

A good way to interview candidates for testing jobs is to give them a simple operation and ask them to enumerate all the things that can possibly go wrong. A classic Microsoft test interview question: how do you test the File Open dialog box? A good tester will be able to rattle off several dozen weird things to test ("file is deleted by another user between the time it is listed in the box and the time you select it and click Open").

OK, so we have one axiom: nothing is as simple as it seems.

There's another axiom in software engineering, which is, always try to reduce risk. One particularly important piece of risk to avoid is schedule risk, when something takes longer than expected. Schedule risk is bad because your boss yells at you, which makes you unhappy. If that's not enough motivation for you, the economic reason why schedule risk is bad is because you decided to do a feature based on information that it would take 1 week. Now that you realize that it has taken 20 weeks to accomplish, that decision might well have been wrong. Perhaps if you knew it was going to take 20 weeks, you would have made a different decision. The more wrong decisions you make, the more likely all those tote bags with your company logo will end up in the liquidator's warehouse while your ex-CEO mopes that "what sucks is, we weren't even successful enough to get mentioned on fuckedcompany when we shut down!"

The combination of nothing-is-as-simple-as-it-seems and reduce-risk can only lead you to one conclusion:

You have to design things before you implement them.

I'm sorry to disappoint you. Yeah, I know, you read Kent Beck and now you think it's OK to not design things before you implement them. Sorry, it's not OK. You can not change things in code "just as easily" as you could change them in the design documents. People say this all the time and it's wrong. "We use high-level tools these days, like Java and XML. We can change things in minutes in the code. Why not design it in code?" My friend, you can put wheels on your mama but that doesn't make her a bus, and if you think you can refactor your wrongly-implemented file-copy function to make it preemptive rather than threaded as quickly as I could write that sentence, you're in deep denial.

Anyway, I don't think Extreme Programming really advocates zero design. They just say "don't do any more design than needed," which is fine. But that's not what people hear. Most programmers are looking for any excuse they can find not to do basic design before implementing features. So they latch onto the "no-design" idea like flies in a bug zapper. Dzzzzzzt! It's one of those weird forms of laziness where you end up doing more work than you would have done otherwise. I'm too lazy to design the feature on paper first, so I just write some code, and then it's not right, so I have to fix it, and I spend more time than I would have otherwise. Or, more commonly, I write some code, and then it's not right, but it's too late, and my product is inferior and I spend until the end of time making up excuses for why it "has to be that way." It's just sloppy and unprofessional.

When Linus Torvalds bashes design, he's talking about huge systems, which have to evolve, or they become Multics. He's not talking about your File Copy code. And when you consider that he had a pretty clear road map of exactly where he was going, it's no wonder Linus doesn't see much value in design. Don't fall for it. Chances are it doesn't apply to you. And anyway, Linus is much smarter than we are, so things that work for him don't work for us normal people.

Incremental design and implementation is good. Frequent releases are fine (although for shrink-wrapped or mass market software, it drives customers crazy, never a good idea -- instead do frequent internal milestones.) Too much formality in design is a waste of time -- I've never seen a project benefit from mindless flowcharting or UMLing or CRCing or whatever the flavor-du-jour is. And those huge 10 million lines-of-code behemoth systems Linus is talking about should evolve, because humans don't really know how to design software on that scale.

But when you sit down to write File Copy, or when you sit down to plan the features of the next release of your software, you gotta design. Don't let the sirens persuade you otherwise.

Come argue with me about this in person at the Cutter Summit , April 29-May 1st, in Cambridge, MA, where I'll be on a panel with Tom DeMarco, Kent Beck, and others.


Our .NET Strategy

Thursday, April 11, 2002

Here are my current thoughts on the gradual migration to .NET development tools at Fog Creek.

The status quo: Most of CityDesk is written in Visual Basic 6.0, with parts written in Visual C++ 6.0. Most of FogBUGZ is written in VBScript for ASP, with parts written in C++. Almost all of our internal tools and our web presense (Fog Shop, Discussions, etc) are written in VBScript for ASP.

Why bother moving to .NET at all? Simply put, it's because .NET appears so far to be one of the most brilliant and productive development environments ever created. ASP.NET really makes it incredibly easy to create useful web applications; over the last couple of days I've been creating some applications we use internally at incredible speed. All the grungy stuff that takes 75% of the time creating web applications with ASP (such as form validation and error reporting) becomes trivial. ASP.NET is as big a jump in productivity over ASP as Java is to C. Wow.

C# has most of the good stuff from Java, with a few small improvements like automatic boxing. Although we have always done what we can to create reasonably object oriented code in ASP and VB6 in the past, switching to C# will be nice.

Finally, the class libraries that ship with .NET are great. The fact that everything, from data access to web development to GUI development, was redesigned means that there is incredible consistency from top to bottom. When you look at the old Win32 API's, it is simply amazing how many different ways there are to get a string back from a function call, for example. Every two years they changed their mind about what a good way is to do this. .NET has cleaned that all up. I love the fact that you can use an ASP.NET calendar widget, which generates HTML that selects a date on a monthly calendar, and be confident that the "date" class you get back from that thing (System.DateTime, I believe) will be the exact same date class that the SQL Server classes expect. In the past you would not believe how much time we wasted doing things like reformatting dates for SQL statements, or converting COleDateTimes to someOtherKindOfDateTime. And finally -- a string data type that works everywhere! Just last week I was writing ATL code and messing around with BSTRs and OLECHARs and char*s and LPSTRs and what a mess that was. Good riddance.

OK, I admit it -- .NET violated the Never Rewrite From Scratch rule. Microsoft got away with it because they had two things. First, they had the world's best language designer, the man who was responsible for 90% of the productivity gains in software development in the last 20 years: Anders Hejlsberg , who gave us Turbo Pascal (thank you!), Delphi (thank you!), WFC (nice try!) and now .NET (smacked the ball outta the park). Second, they put about a zillion engineers on it for about three years, during a period where much of their competition was more-or-less stalled. Remember, just because Microsoft can do something, doesn't mean you can. Microsoft makes their own gravity. Normal rules don't apply to them.

A few dozen people will now proceed to compose angry emails praising some other development environment, or asking why we don't just use Java and get Write Once Run Anywhere (giggle), or Delphi (the talent has left the building. .NET is Delphi 7.0, and 8.0, and 9.0), or Lisp, or whatever. I am getting locked in the Microsoft Trunk! they will say. Regrettably, I don't have time to get into religious discussions right now and I usually find them quite boring. I don't care if Japanese is a better language than English. It just doesn't matter. Let me finish describing our strategy.

First problem: we don't know enough about .NET to write good code. As usual in any development environment there are many ways to do any given thing, and we haven't quite learned the first way, let alone the second way. So the quality of .NET code that we can write is not good enough to ship. Until Bill Vaughn's first ADO book came out, we didn't even know the optimal ways to do basic SQL queries. So our first priority is education, which we will accomplish by doing all future in-house and web-based development in .NET -- basically, all the software that nobody is paying money for. We can migrate parts of the Fog Shop to .NET and certainly use .NET for all kinds of internal stuff. (Today I wrote a FogShop Coupon Generator in ASP.NET. It's kind of messy but it works!)

Second problem: the obese 20 MB CLR (runtime). It's bad enough that about 6 MB of the 8 MB CityDesk download comes from runtimes and data access libraries; we just can't expect every home CityDesk Starter Edition user to download another 20 MB. The hope is that in a year or two or three lots of people will have the CLR from somewhere else (too bad it didn't make it into Windows XP). We'll keep an eye on that; it used to change your user agent; I hope it still does, so we can get statistics.

The bottom line is that neither CityDesk nor FogBUGZ can be ported to .NET today. We will port some future version of CityDesk when the CLR has about 75% penetration. The plan is to:

  1. port existing code and forms using Microsoft's conversion tools
  2. fix problems by brute force until it's working again
  3. create new forms and classes using C#
  4. gradually port old forms and classes to C# whenever they need major work, anyway
  5. many old forms and classes will remain in VB.NET forever (using the ugly backwards compatibility string functions, etc.) as long as they work properly.

FogBUGZ also needs to wait for greater CLR penetration on server computers; we need to survey our customers and find out how bad it would be if FogBUGZ required the CLR.

We have another product in the works which we haven't talked about publically; this one will share a large portion of its code base with FogBUGZ (a subset we are going to name "Dispatcho") so it will remain VBScript/ASP at heart until we port FogBUGZ.

For FogBUGZ/Dispatcho/SecretNewProduct, the plan is:

  1. wait until it's OK to require CLR,
  2. port existing "business logic" classes to C#,
  3. keep current web forms in ASP,
  4. and create new web forms in ASP.NET.

Discuss


Painless Bug Tracking

Wednesday, November 08, 2000

TRS-80 Level-I BASIC could only store two string variables, A$ and B$. Similarly, I was born with only two bug-storing-slots in my brain. At any given time, I can only remember two bugs. If you ask me to remember three, one of them will fall on the floor and get swept under the bed with the dust bunnies, who will eat it.

Keeping a database of bugs is one of the hallmarks of a good software team. I never cease to be amazed at how few teams are actually doing this. One of the biggest incorrect facts that programmers consistently seem to believe is that they can remember all their bugs or keep them on post-it notes.

If I can bend your ear a moment, I'd like to explain a pretty painless way to do bug tracking, in the spirit of my previous articles on painless schedules and painless specs.

First of all, you need a real database. On teams of 2 people writing a little bit of code over the course of a long weekend, it's probably OK to use a text file as the database. Anything larger, and you're going to need a real bug tracking database. There are zillions of bug tracking databases you can buy. (Blatant self-promotion: the one we wrote at Fog Creek Software, called FogBUGZ, is web based, pretty easy to use, and quite powerful, if I may say so myself.)

Let's follow a bug around, for the purpose of illustration, from the moment it's born until someone finally puts it out of its misery. We'll follow the famous Bug 1203. Here's what the bug database shows for that bug:


Painless Functional Specifications - Part 1: Why Bother?

Monday, October 02, 2000

When The Joel Test first appeared, one of the biggest sore points readers reported had to do with writing specs. It seems that specs are like flossing: everybody knows they should be writing them, but nobody does.

Why won't people write specs? People claim that it's because they're saving time by skipping the spec-writing phase. They act as if spec-writing was a luxury reserved for NASA space shuttle engineers, or people who work for giant, established insurance companies. Balderdash. First of all, failing to write a spec is the single biggest unnecessary risk you take in a software project. It's as stupid as setting off to cross the Mojave desert with just the clothes on your back, hoping to "wing it." Programmers and software engineers who dive into code without writing a spec tend to think they're cool gunslingers, shooting from the hip. They're not. They are terribly unproductive. They write bad code and produce shoddy software, and they threaten their projects by taking giant risks which are completely uncalled for.

I believe that on any non-trivial project (more than about 1 week of coding or more than 1 programmer), if you don't have a spec, you will

The most important function of a spec is to design the program. Even if you are working on code all by yourself, and you write a spec solely for your own benefit, the act of writing the spec -- describing how the program works in minute detail -- will force you to actually design the program.

Let's visit two imaginary programmers at two companies. Speedy, at Hasty Bananas Software, never writes specs. "Specs? We don't need no stinkin' specs!" At the same time, Mr. Rogers, over at The Well-Tempered Software Company, refuses to write code until the spec is completely nailed down. These are only two of my many imaginary friends.

Speedy and Mr. Rogers have one thing in common: they are both in charge of backwards compatibility for version 2.0 of their respective products.

Speedy decides that the best way to provide backwards compatibility is to write a converter which simply converts 1.0 version files into 2.0 version files. She starts banging that out. Type, type, type. Clickety clickety clack. Hard drives spin. Dust flies. After about 2 weeks, she has a reasonable converter. But Speedy's customers are unhappy. Speedy's code will force them to upgrade everyone in the company at once to the new version. Speedy's biggest customer, Nanner Splits Unlimited, refuses to buy the new software. Nanner Splits needs to know that version 2.0 will still be able to work on version 1.0 files without converting them. Speedy decides to write a backwards converter and then hook it into the "save" function. It's a bit of a mess, because when you use a version 2.0 feature, it seems to work, until you go to save the file in 1.0 format. Only then are you told that the feature you used half an hour ago doesn't work in the old file format. So the backwards converter took another two weeks to write, and it don't work so nice. Elapsed time, 4 weeks.

Now, Mr. Rogers over at Well-Tempered Software Company (colloquially, "WellTemperSoft") is one of those nerdy organized types who refuses to write code until he's got a spec. He spends about 20 minutes designing the backwards compatibility feature the same way Speedy did, and comes up with a spec that basically says:

  • When opening a file created with an older version of the product, the file is converted to the new format.

The spec is shown to the customer, who says "wait a minute! We don't want to switch everyone at once!" So Mr. Rogers thinks some more, and amends the spec to say:

  • When opening a file created with an older version of the product, the file is converted to the new format in memory. When saving this file, the user is given the option to convert it back.

Another 20 minutes have elapsed.

Mr. Rogers' boss, an object nut, looks at this and thinks something might be amiss. He suggests a different architecture.

  • The code will be factored to use two interfaces: V1 and V2. V1 contains all the version one features, and V2, which inherits from V1, adds all the new features. Now V1::Save can handle the backwards compatibility while V2::Save can be used to save all the new stuff. If you've opened a V1 file and try to use V2 functionality, the program can warn you right away, and you will have to either convert the file or give up the new functionality.

20 more minutes.

Mr. Rogers is grumpy. This refactoring will take 3 weeks, instead of the 2 weeks he originally estimated! But it does solve all the customer problems, in an elegant way, so he goes off and does it.

Total elapsed time for Mr. Rogers: 3 weeks and 1 hour. Elapsed time for Speedy: 4 weeks, but Speedy's code is not as good.

The moral of the story is that with a contrived example, you can prove anything. Oops. No, that's not what I meant to say. The moral of the story is that when you design your product in a human language, it only takes a few minutes to try thinking about several possibilities, revising, and improving your design. Nobody feels bad when they delete a paragraph in a word processor. But when you design your product in a programming language, it takes weeks to do iterative designs. What's worse, a programmer who's just spend 2 weeks writing some code is going to be quite attached to that code, no matter how wrong it is. Nothing Speedy's boss or customers could say would convince her to throw away her beautiful converting code, even though that didn't represent the best architecture. As a result, the final product tends to be a compromise between the initial, wrong design and the ideal design. It was "the best design we could get, given that we'd already written all this code and we just didn't want to throw it away." Not quite as good as "the best design we could get, period."

So that's giant reason number one to write a spec. Giant reason number two is to

When you don't have a spec, all this communication still happens, because it has to, but it happens ad hoc. The QA people fool around with the program willy-nilly, and when something looks odd, they go and interrupt the programmers yet again to ask them another stupid question about how the thing is supposed to work. Besides the fact that this ruins the programmers' productivity, the programmers tend to give the answer that corresponds to what they wrote in the code, rather than the "right answer." So the QA people are really testing the program against the program rather than the program against the design, which would be, um, a little bit more useful.

When you don't have a spec, what happens with the poor technical writers is the funniest (in a sad kind of way). Tech writers often don't have the political clout to interrupt programmers. In many companies, if tech writers get in the habit of interrupting programmers to ask how something is supposed to work, the programmers go to their managers and cry about how they can't get any work done because of these [expletive deleted] writers, and could they please keep them away, and the managers, trying to improve productivity, forbid the tech writers to waste any more of their precious programmers' time. You can always tell these companies, because the help files and the manuals don't give you any more information than you can figure out from the screen. When you see a message on a screen which says

  • Would you like to enable LRF-1914 support?

... and you click "Help", a tragicomic help topic comes up which says something like

  • Allows you to choose between LRF-1914 support (default) or no LRF-1914 support. If you want LRF-1914 support, choose "Yes" or press "Y". If you don't want LRF-1914 support, choose "No" or press "N".

Um, thanks. It's pretty obvious here that the technical writer was trying to cover up the fact that they didn't know what LRF-1914 support is. They couldn't ask the programmer, because (a) they're embarrassed, or (b) the programmer is in Hyderabad and they're in London, or (c) they have been prohibited by management from interrupting the programmer, or any other number of corporate pathologies too numerous to mention, but the fundamental problem is that there wasn't a spec.

Number three giant important reason to have a spec is that without a detailed spec, it's impossible to make a schedule. Not having a schedule is OK if it's your PhD and you plan to spend 14 years on the thing, or if you're a programmer working on the next Duke Nukem and we'll ship when we're good and ready. But for almost any kind of real business, you just have to know how long things are going to take, because developing a product costs money. You wouldn't buy a pair of jeans without knowing what the price is, so how can a responsible business decide whether to build a product without knowing how long it will take and, therefore, how much it will cost? For more on scheduling, read Painless Software Schedules.

A terribly common error is having a debate over how something should be designed, and then never resolving the debate. Brian Valentine, the lead developer on Windows 2000, was famous for his motto "Decisions in 10 minutes or less, or the next one is free."

In too many programming organizations, every time there's a design debate, nobody ever manages to make a decision, usually for political reasons. So the programmers only work on uncontroversial stuff. As time goes on, all the hard decisions are pushed to the end. These are the most likely projects to fail. If you are starting a new company around a new technology and you notice that your company is constitutionally incapable of making decisions, you might as well close down now and return the money to the investors, because you ain't never gonna ship nothing.

Writing a spec is a great way to nail down all those irritating design decisions, large and small, that get covered up if you don't have a spec. Even small decisions can get nailed down with a spec. For example, if you're building a web site with membership, you might all agree that if the user forgets their password, you'll mail it to them. Great. But that's not enough to write the code. To write the code, you need to know the actual words in that email. At most companies, programmers aren't trusted with words that a user might actually see (and for good reason, much of the time). So a marketing person or a PR person or some other English major is likely to be required to come up with the precise wording of the message. "Dear Shlub, Here's the password you forgot. Try not to be so careless in the future." When you force yourself to write a good, complete spec (and I'll talk a lot more about that soon), you notice all these things and you either fix them or at least you mark them with a big red flag.

OK. We're on the same page now. Specs are motherhood and apple pie. I suspect that most people understand this, and my rants, while amusing, aren't teaching you anything new. So why don't people write specs? It's not to save time, because it doesn't, and I think most coders recognize this. (In most organizations, the only "specs" that exist are staccato, one page text documents that a programmer banged out in Notepad after writing the code and after explaining that damn feature to the three hundredth person.)

I think it's because so many people don't like to write. Staring at a blank screen is horribly frustrating. Personally, I overcame my fear of writing by taking a class in college that required a 3-5 page essay once a week. Writing is a muscle. The more you write, the more you'll be able to write. If you need to write specs and you can't, start a journal, create a

If you've never worked in a company that does functional specifications, you may never have seen one. In the next part of this series, I'll show you a short, sample spec for you to check out, and we'll talk about what a good spec needs to have. Read on!


Painless Functional Specifications - Part 2: What's a Spec?

Tuesday, October 03, 2000

(Have you already read part one? If not, that's here .)

This series of articles is about functional specifications, not technical specifications. People get these mixed up. I don't know if there's any standard terminology, but here's what I mean when I use these terms.

  1. A functional specification describes how a product will work entirely from the user's perspective. It doesn't care how the thing is implemented. It talks about features. It specifies screens, menus, dialogs, and so on.
  2. A technical specification describes the internal implementation of the program. It talks about data structures, relational database models, choice of programming languages and tools, algorithms, etc.

When you design a product, inside and out, the most important thing is to nail down the user experience. What are the screens, how do they work, what do they do. Later, you worry about how to get from here to there. There's no use arguing about what programming language to use before you've decided what your product is going to do. In this series, I'm only talking about functional specifications.

I've written a short sample spec which should give you an idea for what a good functional specification looks like. Before we go further, please read the sample spec.

Did you read it?

No you didn't. Go read it now and then come back, so we can talk more about what a good spec should and shouldn't have in it. I'll wait here for you. Thanks.

(waiting patiently...)

Ah, good. You're back.

Here are some of the things I put in every spec.

A disclaimer. Pure self defense. If you put a paragraph saying something like "This spec is not complete", people won't come into your office to bite your head off. As time goes on, when the spec starts to be complete, you can change it to say "this spec is complete, to the best of my knowledge, but if I forgot something, please tell me." Which reminds me, every spec needs:

An author. One author. Some companies think that the spec should be written by a team. If you've ever tried group writing, you know that there is no worse torture. Leave the group writing to the management consulting firms with armies of newly minted Harvard-educated graduates who need to do a ton of busywork so that they can justify their huge fees. Your specs should be owned and written by one person. If you have a big product, split it up into areas and give each area to a different person to spec separately. Other companies think that it's egotistic or not "good teamwork" for a person to "take credit" for a spec by putting their name on it. Nonsense. People should take responsibility and ownership of the things that they specify. If something's wrong with the spec, there should be a designated spec owner, with their name printed right there on the spec, who is responsible for fixing it.

Scenarios. When you're designing a product, you need to have some real live scenarios in mind for how people are going to use it. Otherwise you end up designing a product that doesn't correspond to any real-world usage (like the Cue?Cat). Pick your product's audiences and imagine a fictitious, totally imaginary but totally stereotypical user from each audience who uses the product in a totally typical way. Chapter 9 of my UI design book (available online for free) talks about creating fictional users and scenarios. This is where you put them. The more vivid and realistic the scenario, the better a job you will do designing a product for your real or imagined users, which is why I tend to put in lots of made-up details.

Nongoals. When you're building a product with a team, everybody tends to have their favorite, real or imagined pet features that they just can't live without. If you do them all, it will take infinite time and cost too much money. You have to start culling features right away, and the best way to do this is with a "nongoals" section of the spec. Things we are just not going to do. A nongoal might be a feature you won't have ("no telepathic user interface!") or it might be something more general ("We don't care about performance in this release. The product can be slow, as long as it works. If we have time in version 2, we'll optimize the slow bits.") These nongoals are likely to cause some debate, but it's important to get it out in the open as soon as possible. "Not gonna do it!" as George Sr. puts it.

An Overview. This is like the table of contents for your spec. It might be a simple flowchart, or it might be an extensive architectural discussion. Everybody will read this to get the big picture, then the details will make more sense.

Details, details, details. Finally you go into the details. Most people will skim this until they need to know a particular detail. When you're designing a web-type service, a good way to do this is to give every possible screen a canonical name, and provide a chapter describing each one in utter and mind-numbing detail.

Details are the most important thing in a functional spec. You'll notice in the sample spec how I go into outrageous detail talking about all the error cases for the login page. What if the email address isn't valid? What if the password is wrong? All of these cases correspond to real code that's going to be written, but, more importantly, these cases correspond to decisions that somebody is going to have to make. Somebody has to decide what the policy is going to be for a forgotten password. If you don't decide, you can't write the code. The spec needs to document the decision.

Open Issues. It's OK for the first version of the spec to leave open issues. When I write a first draft, I always have lots of open issues, but I flag them (using a special style so I can search for them) and, if appropriate, discuss the alternatives. By the time the programmers start work, all of these need to be stomped out. (You might think it's OK to just let the programmers start on all the easy stuff, and you'll solve the open issues later. Bad idea. You will have enough problems resolving the new issues that come up when the programmers try to implement the code, without having old open issues around that you knew about in advance and could have solved then. Besides, the way you resolve anything non-trivial may have a major impact on how the code should be written.)

Side notes. While you're writing a spec, remember your various audiences: programmers, testers, marketing, tech writers, etc. As you write the spec you may think of useful factoids that will be helpful to just one of those groups. For example, I flag messages to the programmer, which usually describe some technical implementation detail, as "Technical Notes". Marketing people ignore those. Programmers devour them. My specs are often chock full of "Testing Notes," "Marketing Notes," and "Documentation Notes."

Specs Need To Stay Alive. Some programming teams adopt a "waterfall" mentality: we will design the program all at once, write a spec, print it, and throw it over the wall at the programmers and go home. All I have to say is: "Ha ha ha ha ha ha ha ha!"

This approach is why specs have such a bad reputation. A lot of people have said to me, "specs are useless, because nobody follows them, they're always out of date, and they never reflect the product."

Excuse me. Maybe

To make people's life easier, I don't rerelease the spec daily. I usually keep an up to date version on a server somewhere where the team can use it as a reference. On occasional milestones, I print a copy of the spec with revision marks so that people don't have to reread the whole thing -- they can scan the revision marks to see what changes have been made.

Who should write the specs? Read all about it in Part 3.


Painless Functional Specifications - Part 3: But... How?

Wednesday, October 04, 2000

Now that you've read all about why you need a spec and what a spec has in it, let's talk about who should write them.

Who writes specs?

Let me give you a little Microsoft history here. When Microsoft started growing seriously in the 1980s, everybody there had read The Mythical Man-Month, one of the classics of software management. (If you haven't read it, I highly recommend it.) The main point of that book was that when you add more programmers to a late project, it gets even later. That's because when you have n programmers on a team, the number of communication paths is n(n-1)/2, which grows at O(n2).

So the programmers at Microsoft were worried about how to write bigger and bigger programs, when the prevailing wisdom of the day was that adding programmers just makes things worse.

Charles Simonyi, Microsoft's long time "chief architect", suggested the concept of master programmers. The idea was basically that one master programmer would be responsible for writing all the code, but he or she would rely on a team of junior programmers as "code slaves". Instead of worrying about debugging every function, the master programmer would basically just prototype each function, creating the bare outline, and then throw it to one of the junior programmers to implement. (Of course, Simonyi would be the Master Master Programmer.) The term "Master Programmer" was a bit too medieval, so Microsoft went with "Program Manager."

Theoretically, this was supposed to solve the Mythical Man-Month problem, because nobody has to talk to anyone else -- every junior programmer only talks to the one program manager, and so communication grows at O(n) instead of O(n2).

Well, Simonyi may know Hungarian Notation, but he doesn't know Peopleware. Nobody wants to be a code slave. The system didn't work at all. Eventually, Microsoft discovered that despite the alleged Mythical Man Month, you can still add smart people to a team and get increased output, although at decreasing marginal values. The Excel team had 50 programmers when I was there, and it was marginally more productive than a team of 25 would have been -- but not twice as productive.

The idea of master/slave programming was discredited, but Microsoft still had these people called program managers bouncing around. A smart man named Jabe Blumenthal basically reinvented the position of program manager. Henceforth, the program manager would own the design and the spec for products.

Since then, program managers at Microsoft gather requirements, figure out what the code is supposed to do, and write the specs. There are usually about 5 programmers for every program manager; these programmers are responsible for implementing in code what the program manager has implemented in the form of a spec. A program manager also needs to coordinate marketing, documentation, testing, localization, and all the other annoying details that programmers shouldn't spend time on. Finally, program managers at Microsoft are supposed to have the "big picture" of the company in mind, while programmers are free to concentrate on getting their bits of code exactly right.

Program managers are invaluable. If you've ever complained about how programmers are more concerned with technical elegance than with marketability, you need a program manager. If you've ever complained about how people who can write good code never do a good job of writing good English, you need a program manager. If you've ever complained about how your product seems to drift without any clear direction, you need a program manager.

How do you hire a program manager?

Most companies don't even have the concept of program manager. I think that's too bad. In my time, the groups at Microsoft with strong program managers had very successful products: Excel, Windows 95, and Access come to mind. But other groups (such as MSN 1.0 and Windows NT 1.0) were run by developers who generally ignored the program managers (who weren't very good anyway, and probably deserved to be ignored), and their products were not as successful.

Here are three things to avoid.

1. Don't promote a coder to be a program manager. The skills for being a good program manager (writing clear English, diplomacy, market awareness, user empathy, and good UI design) are very rarely the skills for being a good coder. Sure, some people can do both, but they are rare. Rewarding good coders by promoting them to a different position, one that involves writing English, not C++, is a classic case of the Peter Principle: people tend to be promoted to their level of incompetence.

2. Don't let the marketing people be program managers. No offense, but I think my readers will agree that good marketing people rarely have a good enough grasp of the technology issues to design products.

Basically, program management is a separate career path. All program managers need to be very technical, but they don't have to be good coders. Program managers study UI, meet customers, and write specs. They need to get along with a wide variety of people -- from "moron" customers, to irritating hermit programmers who come to work in Star Trek uniforms, to pompous sales guys in $2000 suits. In some ways, program managers are the glue of software teams. Charisma is crucial.

3. Don't have coders report to the program manager. This is a subtle mistake. As a program manager at Microsoft, I designed the Visual Basic (VBA) strategy for Excel and completely speced out, to the smallest detail, how VBA should be implemented in Excel. My spec ran to about 500 pages. At the height of development for Excel 5.0, I estimated that every morning, 250 people came to work and basically worked off of that huge spec I wrote. I had no idea who all these people were, but there were about a dozen people on the Visual Basic team alone just writing documentation for this thing (not to mention the team writing documentation from the Excel side, or the full time person who was responsible for hyperlinks in the help file.) The weird thing was that I was at the "bottom" of the reporting tree. That's right. NOBODY reported to me. If I wanted people to do something, I had to convince them that it was the right thing to do. When Ben Waldman, the lead developer, didn't want to do something I had speced out, he just didn't do it. When the testers complained that something I had speced was impossible to test completely, I had to simplify it. If any of these people had reported to me, the product wouldn't have been as good. Some of them would have thought that it's inappropriate to second-guess a superior. Other times, I would have just put my foot down and ordered them to do it my way, out of conceit or nearsightedness. As it was, I had no choice but to build consensus. This form of decision making was the best way to get the right thing done.

The final article in my series on specs talks about how to write good specs that people want to read.


Painless Functional Specifications - Part 4: Tips

Sunday, October 15, 2000

OK, we've talked about why you need a spec, what a spec has in it, and who should write them. In this fourth and final part of the series I'll share some of my advice for writing good specs.

The biggest complaint you'll hear from teams that do write specs is that "nobody reads them." When nobody reads specs, the people who write them tend to get a little bit cynical. It's like the old Dilbert cartoon in which engineers use stacks of 4-inch thick specs to build extensions to their cubicles. At your typical big, bureaucratic company, everybody spends months and months writing boring specs. Once the spec is done, it goes up on the shelf, never to be taken down again, and the product is implemented from scratch without any regard to what the spec said, because nobody read the spec, because it was so dang mind-numbing. The very process of writing the spec might have been a good exercise, because it forced everyone, at least, to think over the issues. But the fact that the spec was shelved (unread and unloved) when it was completed makes people feel like it was all a bunch of work for naught.

Also, if your spec never gets read, you get a lot of arguments when the finished product is delivered. Somebody (management, marketing, or a customer) says: "wait a minute! You promised me that there would be a Clam Steamer! Where's the clam steamer?" And the programmers say, "no, actually, if you look on the spec on chapter 3, subchapter 4, paragraph 2.3.0.1, you'll see it says quite explicitly 'no clam steamer.'" But that doesn't satisfy the customer, who is always right, so the grumpy programmers have to go retrofit a clam steamer into the thing (making them even more cynical about specs). Or a manager says, "hey, all the wording on this dialog is too verbose, and there should be an advertisement at the top of every dialog box." And the programmers say, in frustration, "but you approved the spec which precisely listed the layout and contents of every dialog box!" But of course, the manager hadn't actually read the spec, because when he tried, his brain started seeping out through his eye sockets, and anyway, it was interfering with his Tuesday golf game.

So. Specs are good, but not if nobody reads them. As a spec-writer, you have to trick people into reading your stuff, and you should also probably make an effort not to cause any already-too-small brains to leak out through eye-sockets.

Tricking people into reading your stuff is usually just a matter of good writing. But it's not fair of me to just say "be a good writer" and leave it at that. Here are four easy rules that you absolutely must follow to make specs that get read.

Rule 1: Be Funny

Yep, rule number one in tricking people into reading your spec is to make the experience enjoyable. Don't tell me you weren't born funny, I don't buy it. Everybody has funny ideas all the time, they just self-censor them because they think that it's "unprofessional." Feh. Sometimes you have to break the rules.

If you read the volumes of garbage I've written on this web site, you'll notice that there are a few lame attempts at being funny scattered throughout. Just four paragraphs ago I was making a gross body-fluid joke and making fun of managers for playing golf. Even though I'm not really that funny, I still try pretty hard, and even the act of flailing around trying to be funny is in itself amusing, in a sad-clown sort of way. When you're writing a spec, an easy place to be funny is in the examples. Every time you need to tell a story about how a feature works, instead of saying:

  • The user types Ctrl+N to create a new Employee table and starts entering the names of the employees.

write something like:

  • Miss Piggy, poking at the keyboard with a eyeliner stick because her chubby little fingers are too fat to press individual keys, types Ctrl+N to create a new Boyfriend table and types in the single record "Kermit."

If you read a lot of Dave Barry, you'll discover that one of the easiest ways to be funny is to be specific when it's not called for. "Scrappy pugs" are funnier than "dogs." "Miss Piggy" is funnier than "the user". Instead of saying "special interests," say "left-handed avocado farmers." Instead of saying "People who refuse to clean up after their dogs should be punished," say that they should be "sent to prisons so lonely that the inmates have to pay spiders for sex."

Oh, and, by the way, if you think that it's unprofessional to be funny, then I'm sorry, but you just don't have a sense of humor. (Don't deny it. People without senses of humors always deny it. You can't fool me.) And if you work in a company where people will respect you less because your specs are breezy, funny, and enjoyable to read, then go find another company to work for, because life is just too damn short to spend your daylight hours in such a stern and miserable place.

Rule 2: Writing a spec is like writing code for a brain to execute

Here's why I think that programmers have trouble writing good specs.

When you write code, your primary audience is the compiler. Yeah, I know, people have to read code, too, but it's generally very hard for them. For most programmers it's hard enough to get the code into a state where the compiler reads it and correctly interprets it; worrying about making human-readable code is a luxury. Whether you write:

void print_count( FILE* a, char  *  b, int c ){
    fprintf(a, "there are %d %s\n", c, b);}

main(){ int n; n =
10; print_count(stdout, "employees", n) /* code
deliberately obfuscated */ }

or

printf("there are 10 employees\n");

you get the same output. Which is why, if you think about it, you tend to get programmers who write things like:

Assume a function AddressOf(x) which is defined as the mapping from a user x, to the RFC-822 compliant email address of that user, an ANSI string. Let us assume user A and user B, where A wants to send an email to user B. So user A initiates a new message using any (but not all) of the techniques defined elsewhere, and types AddressOf(B) in the To: editbox.

This could also have been speced as:

Miss Piggy wants to go to lunch, so she starts a new email and types Kermit's address in the "To:" box.

Technical note: the address must be a standard Internet address (RFC-822 compliant.)

They both "mean" the same thing, theoretically, except that the first example is impossible to understand unless you carefully decode it, and the second example is easy to understand. Programmers often try to write specs which look like dense academic papers. They think that a "correct" spec needs to be "technically" correct and then they are off the hook.

The mistake is that when you write a spec, in addition to being correct, it has to be understandable, which, in programming terms, means that it needs to be written so that the human brain can "compile" it. One of the big differences between computers and human brains is that computers are willing to sit there patiently while you define the terms that you want to use later. But humans won't understand what you're talking about unless you motivate it first. Humans don't want to have to decode something, they just want to read it in order and understand it. For humans, you have to provide the big picture and then fill in the details. With computer programs, you start at the top and work your way to the bottom, with full details throughout. A computer doesn't care if your variable names are meaningful. A human brain understands things much better if you can paint a vivid picture in their mind by telling a story, even if it's just a fragment of a story, because our brains have evolved to understand stories.

If you show a chess board, in the middle of a real game of chess, to an experienced chess player for even a second or two, they will instantly be able to memorize the position of every piece. But if you move around a couple of pieces in nonsensical ways that couldn't happen in normal play (for example, put some pawns on the first row, or put both black bishops on black squares), it becomes much, much harder for them to memorize the board. This is different from the way computers think. A computer program that could memorize a chess board could memorize both possible and impossible layouts with equal ease. The way the human brain works is not random access; pathways tend to be strengthened in our brains and some things are just easier to understand than other things because they are more common.

So, when you're writing a spec, try to imagine the person you are addressing it to, and try to imagine what you're asking them to understand at every step. Sentence by sentence, ask yourself if the person reading this sentence will understand it at a deep level, in the context of what you've already told them. If some members of your target audience don't know what RFC-822 is, you either have to define it, or, at the very least, bury the mention of RFC-822 in a technical note, so that the executive-types who read the spec won't give up and stop reading the first time they see a lot of technical jargon.

Rule 3: Write as simply as possible

Don't use stilted, formal language because you think it's unprofessional to write in simple sentences. Use the simplest language you can.

People use words like "utilize" because they think that "use" looks unprofessional. (There's that word "unprofessional" again. Any time somebody tells you that you shouldn't do something because it's "unprofessional," you know that they've run out of real arguments.) In fact I think that many people think that clear writing means that something is wrong.


Painless Software Schedules

Wednesday, March 29, 2000

Last October, the Northeast US was plastered with ads for something called Acela, a new express train running from Boston to Washington. With TV ads, billboards, and posters everywhere, you'd think that it would have created some demand for Amtrak's new express service.

Well, maybe. Amtrak didn't get a chance to find out. Acela was delayed, and delayed again, so the marketing campaign played out while Acela service wasn't even available. Which reminded me of something I heard a marketing manager say when his product got a rave review one month before it went on sale: "Great publicity! Too bad you can't buy the dang thing!"

Testosterone-crazed game companies like to brag on their web sites that the next game will ship "when it's ready". Schedule? We don't need no stinkin' schedule! We're cool game coders! Most companies don't get that luxury. Ask Lotus. When they first shipped 123 version 3.0, it required an 80286 computer, which wasn't very common then. They delayed the product by 16 months while they worked to shoehorn it into the 640K memory limit of the 8086. By the time they were done, Microsoft had a 16 month lead in developing Excel, and, in a great karmic joke, the 8086 was obsolete anyway!

As I write this, Netscape's 5.0 web browser is almost two years late. Partially, this is because they made the suicidal mistake of throwing out all their code and starting over: the same mistake that doomed Ashton-Tate, Lotus, and Apple's MacOS to the recycle-bins of software history. Netscape has seen its browser share go from about 80% to about 20% during this time, all the while it could do nothing to address competitive concerns, because their key software product was disassembled in 1000 pieces on the floor and was in no shape to drive anywhere. That single bad decision, more than anything else, was the nuclear bomb Netscape blew itself up with. (Jamie Zawinski's world-famous tantrum has the details).

So, you have to make a schedule. This is something almost no programmer wants to do. In my experience, the vast majority just try to get away with not making a schedule at all. Of the few that make a schedule, most are only doing it because their boss made them do it, halfheartedly, and nobody actually believes the schedule except for upper management, which simultaneously believes that "no software project is ever on time" and in the existence of UFOs.

So why doesn't anybody make a schedule? Two key reasons. One, it's a real pain. Two, nobody believes that it's worth anything. Why go to all the trouble working on a schedule if it's not going to be right? There is a perception that schedules are consistently wrong, and only get worse as time goes on, so why suffer for naught?

Here's a simple, painless way to make schedules that are actually correct.

1) Use Microsoft Excel. Don't use anything fancy like Microsoft Project. The trouble with Microsoft Project is that it assumes that you want to spend a lot of time worrying about dependencies. A dependency is when you have two tasks, one of which must be completed before the next one can begin. I've found that with software, the dependencies are so obvious that it's just not worth the effort to formally keep track of them.

Another problem with Project is that it assumes that you're going to want to be able to press a little button and "rebalance" the schedule. Inevitably, this means that it's going to rearrange things and reassign things to different people. For software, this just doesn't make sense. Programmers are not interchangeable. It takes seven times longer for John to fix Rita's bug than for Rita to fix Rita's bug. And if you try to put your UI programmer on a WinSock problem, she'll stall and waste a week getting up to speed on WinSock programming. The bottom line is that Project is designed for building office buildings, not software.

2) Keep it Simple. The standard format I use for schedules is so simple you can memorize it. You start with just seven columns:

If you have several developers, you can either keep a separate sheet for each developer, or you can make a column with the name of the developer working on each task.

3) Each feature should consist of several tasks. A feature is something like adding a spell checker to your program. Adding a spell checker consists of quite a few discrete tasks that the programmer has to do. The most important part of making a schedule is making this list of tasks. Thus the cardinal rule:

4) Only the programmer who is going to write the code can schedule it. Any system where management writes a schedule and hands it off to programmers is doomed to fail. Only the programmer who is going to do the work can figure out what steps they will need to take to implement that feature. And only the programmer can estimate how long each one will take.

5) Pick very fine grained tasks. This is the most important part to making your schedule work. Your tasks should be measured in hours, not days. (When I see a schedule measured in days, or even weeks, I know it's not real). You might think that a schedule with fine grained tasks is merely more precise. Wrong! Very wrong! When you start with a schedule with rough tasks and then break it down into smaller tasks, you will find that you get a different result, not just a more precise one. It is a completely different number. Why does this happen?

When you have to pick fine grained tasks, you are forcing yourself to actually figure out what steps you are going to have to take. Write subroutine foo. Create dialog such and such. Read the wawa file. These steps are easy to estimate, because you've written subroutines, created dialogs, and read wawa files before.

If you are sloppy, and pick big "chunky" tasks ("implement grammar correction"), then you haven't really thought about what you are going to do. And when you haven't thought about what you're going to do, you just can't know how long it will take.

As a rule of thumb, each task should be from 2 to 16 hours. If you have a 40 hour (one week) task on your schedule, you're not breaking it down enough.

Here's another reason to pick fine grained tasks: it forces you to design the damn feature. If you have a hand-wavy feature called "Internet Integration" and you schedule 3 weeks for it, you are doomed, buddy. If you have to figure out what subroutines you're going to write, you are forced to pin down the feature. By being forced to plan ahead at this level, you eliminate a lot of the instability in a software project.

6) Keep track of the original and current estimate. When you first add a task to the schedule, estimate how long it's going to take in hours and put that in both the Orig[inal] Est[imate] and Curr[ent] Est[imate] columns. As time goes on, if a task is taking longer (or shorter) than you thought, you can update the Curr Est column as much as you need. This is the best way to learn from your mistakes and teach yourself how to estimate tasks well. Most programmers have no idea how to guess how long things will take. That's okay. As long as you are continuously learning and continuously updating the schedule as you learn, the schedule will work. (You may have to cut features or slip, but the schedule will still be working correctly, in the sense that it will constantly be telling you when you have to cut features or slip). I've found that most programmers become very good schedulers with about one year of experience.

When the task is done, the Curr Est and Elapsed fields will be the same, and the Remain field will recalc to 0.

7) Update the elapsed column every day. You do not really have to watch your stopwatch while you code. Right before you go home, or go to sleep under the desk if you're one of those geeks, pretend you've worked for 8 hours (ha!), figure out which tasks you've worked on, and sprinkle about 8 hours in the elapsed column accordingly. The remaining time field is then calculated automatically by Excel.

At the same time, update the Curr Est column for those tasks to reflect the new reality. Updating your schedule daily should only take about two minutes. That's why this is the Painless Schedule Method -- it's quick and easy.

8) Put in line items for Vacations, Holidays, etc. If your schedule is going to take about a year, each programmer will probably take 10 to 15 days of vacation. You should have a feature in your schedule called vacations, one for holidays, and anything else that consumes people's time. The idea is that the ship date can be calculated by adding up the remaining time column and dividing by 40 -- that's how many weeks of work are left, including everything.

9) Put debugging time into the schedule! Debugging is the hardest to estimate. Think back to the last project you worked on. Chances are, debugging took from 100% - 200% of the time it took to write the code in the first place. This has to be a line item in the schedule, and it will probably be the largest line item.

Here's how it works. Let's say a developer is working on wawa. The Orig Est was 16 hours, but so far it has taken 20 hours and it looks like it needs another 10 hours of work. So the developer enters 30 under Curr Est and 20 under elapsed.

At the end of the milestone, all these "slips" have probably added up to quite a bit. Theoretically, to accommodate these slips, we have to cut features in order to ship on time. Luckily, the first feature we can cut is this great big feature called Buffer which has lots o' hours already allocated for it.

In principle, developers debug code as they write it. A programmer should never, ever work on new code if they could instead be fixing bugs. The bug count must stay as low as possible at all times, for two reasons:

1) It's easier to fix bugs the same day you wrote the code. It can be very hard and time-consuming to fix bugs a month later when you've forgotten exactly how the code works.

2) Fixing bugs is like doing science. It is impossible to estimate when you will make the discovery and solve the bug. If there are only one or two outstanding bugs at any given time, it's easy to estimate when the product will ship, because there's not much un-estimateable science in your future. On the other hand, if there are hundreds or thousands of outstanding bugs, it is impossible to predict when they will all be fixed.

If developers always fix bugs as they go along, what's the point in having a debugging line item? Well, even if you try to fix every bug as you go along, at the end of every milestone, there is inevitably a lot of bug fixing when testers (internal and beta) find the really hard bugs.

10) Put integration time into the schedule. If you have more than one programmer, inevitably, there will be stuff that two programmers do that is inconsistent and needs to be reconciled. They might both implement dialog boxes for similar things that are unnecessarily inconsistent. Somebody will have to go through all the menus, keyboard accelerators, toolbar tools, etc., cleaning up and organizing all the new menu items that everybody has been adding willy-nilly. There will be compiler errors that show up as soon as two people check in code. This has to be fixed, and it should be a line item on the schedule.

11) Put buffer into the schedule. Things tend to run over. There are two important kinds of buffer you might want to consider. First: buffer to account for tasks that took longer than originally estimated. Second: buffer to account for tasks that you didn't know you would have to do, usually because management has decided that implementing wawa is SUPER IMPORTANT and cannot be left out of the next release.

You may be surprised to find that vacations, holidays, debugging, integration, and buffer time add up to more than the actual tasks do. If you're surprised by this, you haven't been programming for long, have you? Ignore these at your peril.

12) Never, ever let managers tell programmers to reduce an estimate. Many rookie software managers think that they can "motivate" their programmers to work faster by giving them nice, "tight" (unrealistically short) schedules. I think this kind of motivation is brain-dead. When I'm behind schedule, I feel doomed and depressed and unmotivated. When I'm working ahead of schedule, I'm cheerful and productive. The schedule is not the place to play psychological games.

If your manager makes you reduce an estimate, here's what to do. Create a new column in the schedule called Rick's Estimate (assuming your name is Rick, of course.) Put your estimate in there. Let your manager do whatever she wants with the Curr Est column. Ignore your manager's estimates. When the project is done, see who was closer to reality. I've found that just threatening to do this works wonders, especially when your manager realizes that they've just gotten into a contest to see how slowly you can work!

Why do inept managers try to get programmers to reduce estimates?

When the project begins, the technical managers go off, meet with the business people, and come up with a list of features which they think would take about 3 months, but would really take 9. When you think of writing code without thinking about all the steps you have to take, it always seems like it will take n time, when in reality it will probably take more like 3n time. When you do a real schedule, you add up all the tasks and realize that the project is going to take much longer than originally thought. Reality sinks in.

Inept managers try to address this by figuring out how to get people to work faster. This is not very realistic. You might be able to hire more people, but they need to get up to speed and will probably be working at 50% efficiency for several months (and dragging down the efficiency of the people who have to mentor them). Anyway, in this market, adding good programmers is going to take 6 months.

You might be able to get 10% more raw code out of people temporarily at the cost of having them burn out 100% in a year. Not a big gain, and it's a bit like eating your seed corn.

You might be able to get 20% more raw code out of people by begging everybody to work super hard, no matter how tired they get. Boom, debugging time doubles. An idiotic move that backfires in a splendidly karmic way.

But you can never get 3n from n, ever, and if you think you can, please email me the stock ticker of your company so I can short it.

13) A schedule is like wood blocks. If you have a bunch of wood blocks, and you can't fit them into a box, you have two choices: get a bigger box, or remove some blocks. If you thought you could ship in 6 months, but you have 12 months on the schedule, you are either going to have to delay shipping, or find some features to delete. You just can't shrink the blocks, and if you pretend you can, then you are merely depriving yourself of a useful opportunity to actually see into the future by lying to yourself about what you see there.

And you know, the other great byproduct of keeping schedules like this is that you are forced to delete features. Why is this good? Suppose you have two features: one which is really useful and will make your product really great (example: tables in Netscape 2.0), and another one which is really easy and which the programmers would love to code (example: the BLINK tag), but which serves no useful or marketing purpose.

If you don't make a schedule, the programmers will do the easy/fun feature first. Then they'll run out of time, and you will have no choice but to slip the schedule to do the useful/important feature.

If you do make a schedule, even before you start working, you'll realize that you have to cut something, so you'll cut the easy/fun feature and just do the useful/important feature. By forcing yourself to chose some features to cut, you wind up making a more powerful, better product with a better mix of good features that ships sooner.

I remember working on Excel 5. Our original feature list was huge and would have gone way over schedule. Oh my! we thought. Those are all super important features! How can we live without a macro editing wizard?

As it turns out, we had no choice, and we cut what we thought was "to the bone" to make the schedule. Everybody felt unhappy about the cuts. To assuage our feelings, we simply told ourselves that we weren't cutting the features, we were simply deferring them to Excel 6, since they were less important.

As Excel 5 was nearing completion, I started working on the Excel 6 spec with a colleague, Eric Michelman. We sat down to go through the list of "Excel 6" features that had been cut from the Excel 5 schedule. We were absolutely

Appendix: Things you should know about Excel

One of the reasons that Excel is such a great product for working on software schedules is that the only thing most Excel programmers use Excel for is maintaining their software schedules! (Not many of them are running business what-if scenarios... these are programmers, here!)

Shared Lists Using the File/Shared Lists command allows everyone to open the file at the same time and edit things at the same time. Since your whole team should be updating the schedule constantly, this really helps.

Auto Filter This is a great way to filter the schedule so that, for example, you only see all of the features that are assigned to you. Combined with Auto Sort, you can see all of the features assigned to you in order of priority which is effectively your "to do" list. Cooooool!

Pivot Tables This is a great way to see summaries and crosstabulations. For example, you can make a chart showing the remaining hours for each developer for each priority. Pivot Tables are like sliced bread and chocolate milkshakes. You gotta learn how to use them because they make Excel a million times more powerful.

The WORKDAY Function from Excel's Analysis Toolpak is a great way to get calendar dates out of a Painless Schedule.

Some Q&A on this article: Juggling Tasks in Excel

Safari Software ships a free Excel template, MasterList-XL for task management based on these principles.


Passport Responses

Friday, July 28, 2000

My article Does Issuing Passports Make Microsoft a Country? seem to have hit a nerve. Here are some quotes from the responses I received. I can't resist mentioning one fact: out of the dozens of emails I received, only one supported Passport, and that was from a Microsoft employee. Which just goes to show how powerful the Microsoft Reality Distortion Field gets in Redmond.

Microsoft controls PassPort and.. they control Frontpage, arguably the most popular web management software. How much time before they put "Passport extensions" into Frontpage that will make your and my website part of the PassPort network? It's not just Microsoft sites, they'll know all about you if you visit competitors, mom-n-pop shops, porn sites, etc etc.

-- Jacob Levy

That's scary stuff. Enough so, that I've added an entry to my local DNS server which says *.passport.com is a CNAME for 127.0.0.1. Interestingly enough, most MS sites quit working, but I'm pretty sure I'm off their radar now.

-- Dave Polaschek, Polaschek Computing

What Microsoft is doing is what every ad banner network has been doing for ages: spying on you. Ever since the cookie was integrated into most browsers, there has been zero privacy on the net, ethical or not.

-- Nick Bauman, WebHelp.com

Following on from your M$ Passport article, I have not seen anybody talking/writing about the fact that the M$ cookie-cutter "initiative" that has got DoubleClick and others so riled, is just another stick to lever business onto Passport, because it will be one of the few ways to achieve what cookies do (and worse obviously) without them.

-- Robin Benson

The new MSN 6 beta browser (downloadable from msn.com as "MSN review 1") does in fact eliminate this step. To use the browser you have to enter your Hotmail ID and password. Then you are treated to a flashing white bar and a message that says "signing in...". After that, access to MSN, Hotmail etc is seamless... no signing in or out.

-- Prasenjeet Dutta

Do you think the alternative: typing all of your information every time you log into a site and having so many passwords to remember is acceptable?

Do you use multiple credit cards so that no one credit card company knows your purchase habits?

Wouldn?euro;™t you prefer targeted ads (e.g. houses for sale in Jerusalem) to the random ones?euro;”given that you?euro;™re seeing ads anyway?

Single identity is the #1 feature of AOL and Amazon (why else would you pay more for toys from Amazon?)

It?euro;™s silly to suggest the theoretical concern that one day the system will try to charge you more if you have money?euro;”(a) first of all, that?euro;™s true today anyway (if you buy an item from a high-end retailer targeted at wealthier folks you will pay more than if you bought the same thing at Target), but (b) the web makes it so easy to compare prices that Expedia couldn?euro;™t get away with your scenario. And, passport requires a log-in, so it?euro;™s easier to hide your Bankruptcy book purchase from passport than it is to the credit card companies.

Someone has to solve this problem. If you think you know a better way, then that would be a much more interesting article?euro;?/p>

-- An [anonymous] 10 year Microsoft veteran


Picking a Ship Date

Tuesday, April 09, 2002

One of the best reasons to make a detailed schedule is because it gives you an excuse to cut features. If there's no way to make the ship date and implement Bob's Singalong MP3 Chat feature, it's easy to cut that feature without making Bob feel bad.

So my basic rule for software release cycles is:

  1. Set a ship date, which might as well be arbitrary
  2. Make a list of features and sort them out by priority
  3. Cut low-priority features every time you slip so as to make the date.

If you do this well, you'll soon discover that you don't regret the features that you cut. They're usually kind of dumb. If they were so important, you can do them next time. It's like editing. If you ever want to write a brilliant 750 word essay, start by writing a 1500 word essay and then edit.

One way to screw this up, by the way, is forgetting to do features in order of priority. If you aren't careful, your programmers are going to do features in order of fun, and you won't be able to ship or cut features because all the programmers were busy writing Karaoke Recalc before they even got the menus to work, and now here it is, six months after the day you wanted to ship, and you have one hell of an easter egg but no functionality.

So the obvious question is: how do you pick a ship date?

It's conceivable that you have outside constraints. The stock market is switching from fractions to decimals on such-and-such a date, and if you haven't got the new software ready, your firm will be forced out of business and you personally will be taken out behind the loading docks and shot in the head. Or maybe a new version of the Linux kernel is coming out soon with yet another all-new system to implement packet filtering; all your customers are getting it; and your existing application won't run on it. OK, for those people, your ship date is easy to figure out. You can stop reading this article now. Go cook a nice dinner for your loved ones.

Bye, now!

But how should the rest of us pick a ship date?

There are three approaches you could take.

  1. Frequent Small Releases. This is the Extreme Programming approach, and is most appropriate for small-team projects with a small number of customers, such as in-house IT development.
  2. Every 12 to 18 months. This is typical of shrinkwrapped software products, desktop applications, etc., where you have larger teams and thousands or millions of customers.
  3. Every 3-5 years. This is typical of huge software systems and platforms that are worlds unto themselves. Operating systems, .Net, Oracle, and for some reason Mozilla fall into this category. They often have thousands of developers (VS.Net had 50 people on the installer team) and enormously complex interactions to other shipping software which can't be allowed to break.

Here are some of the things you need to think about when deciding how often to release software.

Short releases get customer feedback fast. Sometimes the best way to work with a customer is to show them code, let them try it out, and immediately incorporate their feedback into a build which you give them the very next day. You won't waste a year developing a complicated system with lots of features that nobody uses, because you'll be so busy doing things that customers are requesting right now. If you have a small number of customers, prefer frequent small releases. The size of each release should be the minimum chunk that does something useful.

Several years ago I was assigned to develop a web content management system for MTV. The requirements called for a database-backed system with templates, and a complete workflow system that allowed unpaid MTV stringers at colleges across the country to input information about clubs, record stores, radio stations, and concerts. "How are you building the site now?" I asked.

"Oh, we just do it all manually with BBEdit," they told me. "Sure, there are thousands of pages, but BBEdit has a really good global-find-and-replace function..."

I figured the whole system would take six months to deliver. "But let me suggest something else. Let's get the templating stuff working first. I can get you that in 3 months and it will save tons of manual work right away. Once that's working we'll start in on the workflow component; in the meantime you can continue to do workflow with email."

They agreed. It sounded like a great idea. Guess what? Once I delivered the templating feature, they realized that they didn't really need workflow that much. And the templating turned out to be useful for lots of other web sites which didn't need workflow, either. So we never built workflow, saving three months which I used to enhance the templating feature which turned out to be more useful.

Some types of customers don't appreciate being "guinea pigs" in this fashion. Generally, people who buy off-the-shelf software don't want to be part of a Grand Development Experiment; they want something that anticipates their needs. As a customer, the only thing better than getting feature requests done quickly is getting them instantaneously because they're already in the product, because it was designed thoughtfully and extensively usability- and beta-tested before being inflicted on the world. If you have (or want) a large number of paying customers, prefer less frequent releases.

If you ship an anemic commercial program just to get something out the door so you can "start listening to customers," what you'll hear those customers saying is "it doesn't do very much," which you might think is OK. Hey, it's 1.0. But then if you release 2.0 four months later, everybody's going to think, "That feeble program? What am I, supposed to keep evaluating it every four months just to see if it's gotten better yet?!" And in fact five years down the line, people will still remember their first impression of 1.0, and it will be almost impossible to get them to reevaluate. Think about what happened to poor Marimba. They launched their company with infinite VC in the days of hyper-Java-hype, having lured the key developers from the Java team at Sun. They had a CEO, Kim Polese, who was brilliant at public relations; when she was marketing Java she had Danny Hillis making speeches about how Java was the next step in human evolution; George Gilder wrote these breathless articles about how Java was going to completely upturn the very nature of human civilization. Compared to Java, we were to believe, monotheism, for example, was just a wee blip. Polese is that good. So when Marimba Castanet launched it probably had more unearned hype than any product in history, but the developers had only been working on it for a total of ... four months. We all downloaded it and discovered that -- tada! -- it was a list box that downloaded software. (What do you expect from four months of development?) Big whoop. The disappointment was so thick you could cut it with butter. And here it is, six years later, ask anybody what Castanet is and they'll tell you it's a list box that downloads software. Hardly anyone bothered to reevaluate it, and Marimba has had six years to write code; I'm sure it's just the coolest thing now but, honestly, who has time to find out? Let me tell you a little secret: our strategy for CityDesk is to avoid massive PR until 2.0 is out. That's the version that we want everybody on earth to get their first impressions from. In the meantime we'll do quiet guerilla marketing, and anybody who finds it will discover that it's a completely spiffy program that solves a lot of problems, but Arnold Toynbee won't have to rewrite anything.

With most commercial software, you'll discover that the process of designing, prototyping, integrating, fixing bugs, running a full alpha and beta cycles, creating documentation, and so forth takes 6-9 months. In fact if you try to do a full release every year, you only have time for about 3 months worth of new code development. Software that is upgraded annually usually doesn't feel like it has enough new features to justify the upgrade. (Corel PhotoPaint and Intuit Quickbooks are particularly egregious examples of this; they have a new "major" version every year which is rarely worth buying). As a result many people have learned by now to skip every other release. You don't want your customers getting in that habit. If you stretch out your schedule to 15 or 18 months between releases, you get 6 months of new features instead of 3 months worth, which makes your upgrade a lot more compelling.

OK, if 15 months is so good, wouldn't 24 months be better? Maybe. Some companies can get away with it, if they are major leaders in their category. PhotoShop seems to get away with it. But as soon as an application starts to feel old, people stop buying it because they are expecting that new version Any Day Now. This can cause serious cash flow problems for a software business. And of course you may have competitors nipping at your heels.

For large platform software -- operating systems, compilers, web browsers, DBMSs -- the hardest part of the development process is maintaining compatibility with thousands or millions of existing applications or hardware. When a new version of Windows comes out, you very rarely hear about backwards compatibility problems. The only way they can achieve this is with insane amounts of testing that make the construction of the Panama Canal seem like a weekend do-it-yourself project. Given the typical 3 year cycle between major Windows releases, almost all of that time is spent in the boring integration and testing phase, not writing new features. Releasing new versions any more often than that is just not realistic. And it would drive people crazy. Third party software and hardware developers would simply revolt if they had to test against lots of little incremental releases of an operating system. For Systems With Millions of Customers and Millions of Integration Points, Prefer Rare Releases. You can do it like Apache: one release at the beginning of the Internet Bubble, and one release at the end. Perfect.

If you have a lot of validation and unit tests, and if you write your software carefully, you may get to the point where any daily build is almost high enough quality to ship. This is certainly something to strive for. Even if you're planning the next release for three years from now, the competitive landscape may suddenly change and there may be a good reason to do a quick interim release to react to a competitor. When your wife is about to give birth, it's not really a good idea to take apart your car's engine. Instead build a new one on the side and don't hook it up until it's perfect.

But don't overestimate what you can accomplish by keeping high quality daily builds. Even if you are permanently at zero bugs, if your software has to run In The Wild, you're never going to find all the compatibility bugs and Windows 95 bugs and It-Doesn't-Work-With-Large-Fonts-Turned-On bugs until you do a few betas, and there is no realistic way to do a beta cycle in less than 8 weeks.

One final thought. If your software is delivered as a service over the web, like ebay or PayPal, theoretically there's nothing to stop you from frequent small releases, but this might not be the best thing to do. Remember the cardinal rule of usability: an application is usable if it behaves the way that the user expected it to behave. If you keep changing things around every week, it won't be so predictable, so it won't be so usable. (And don't think you can get around this by having one of those annoying screens with a paragraph of text saying Warning! The UI has changed! Nobody reads those.) From a usability perspective a better approach would probably be less frequent releases that include a bunch of changes all at once, where you make an effort to change the visuals of the whole site so that it all looks weird and users intuit that things have changed a lot and they have to be careful.


Platforms

Friday, August 30, 2002

Most software developers, including Fog Creek Software, are perfectly happy just to write software applications. You know, programs that do something or solve some particular problem. But the brave among us want to change the world more significantly, and they choose to work on platforms: big giant slabs of software that don't quite do anything out of the box, but which enable a whole world of new and interesting applications. So they write operating systems, or DBMSs, or language runtimes like Java, and they hope to attract independent software developers to create the great new applications that their platforms enable.

Almost by definition, an operating system is a platform. Many platforms live on top of operating systems, such as the Java Runtime. And don't forget that Windows didn't start out as an OS, it started out as a program you ran on DOS which (out of the box) didn't do much of anything, but enabled software developers to create GUI applications for inexpensive Intel boxes.

It's really, really important to figure out if your product is a platform or not, because platforms need to be marketed in a very different way to be successful. That's because a platform needs to appeal to developers first and foremost, not end users.

I just had the good fortune to read an advance copy of Rick Chapman's excellent new book on stupidity in the software industry. (Prediction: best seller). Being an analytical kind of guy, I look for common themes. One of the biggest themes in software industry failures is a platform vendor that didn't understand that they were a platform vendor, so they alienated their key constituency: the developers.

Example: NetWare took so long to release reasonable tools for creating NLMs that when Unix and Windows NT came along with superior and cheaper development tools, they swept away the mindshare of server software developers.

Example: Apple has spent decades making life miserable for their developers. Every new OS for almost 20 years required tweaks and changes to application code. If you got too successful, Apple competed against you (although sometimes they had a hand puppet called Claris compete against you so they could pretend it wasn't them.)

Example: Developing software for OS/2 1.0 required an investment of $3000 for the SDK, and you had to write all your own printer drivers if printing was important for you. Printing was important, so OS/2 had no applications.

But the counter-examples are just as interesting:

Example: The first versions of Windows included a freely redistributable runtime so that if you wrote a Windows application, you could sell it to anyone with DOS, you weren't limited to the few weirdo dorks (me!) who bought Windows 1.0.

Example: Despite the mistakes that Sun made with Java, the runtime was always free and good Java tools were cheap or free, too. No other development platform became so predominant so quickly (even Visual Basic, the top selling computer language of all time, took years to ramp up.)

If you want a platform to be successful, you need massive adoption, and that means you need developers to develop for it. The best way to kill a platform is to make it hard for developers to build on it. Most of the time, this happens because platform companies either don't know that they have a platform (they think it's an application) or they get greedy (they want all the revenue for themselves.)

Greedy platform companies can't stand the idea of all kinds of unwashed riffraff making money off of their platform, so they make it darn near impossible for anyone to develop for it. Probably the most spectacular failure illustrating this was IBM's PS/2, with its huge portfolio of proprietary technologies, such as the new Microchannel architecture designed to insure that only IBM could make expansion cards. This is, of course, monumentally shortsighted. Nobody wanted PS/2s because, uh, add-in cards weren't available and they were too expensive when they were. As a platform vendor, you're only as successful as the people who build on you.

A more subtle problem is when platform vendors don't think they have a platform, they think they have an application. To illustrate this I'm once again going to have to pick on Groove.

"Why do you keep picking on Groove, Joel?" Three reasons:

  • They have an interesting architecture that provides important platform functionality which I could really use in my own products,
  • They make it impossible (or at least unrealistic) for ISVs to build on their platform, thus dooming themselves to oblivion, either out of greed or because they think Groove is an application, not a platform,
  • and Groove inventor Ray Ozzie has a weblog, so he can answer me if he thinks I'm off the mark. (He did.)

Here's how I noticed the Groove problem. I've got a product, CityDesk, for simple desktop web content management. Weblogs, company sites, small organizations, etc. -- people who need content management but can't afford the big systems, don't control a server anywhere, or just can't be bothered to mess with installing perl scripts on a server.

As a 1.0 product we've got our share of weaknesses. One of the big ones is that people want to collaborate on CityDesk sites over the Internet. A reasonable request. For the next major release, we have to do something about this limitation. Basically, we've got two choices. The traditional choice would be to do something client-server: make a CityDesk server you can install somewhere so that everyone can collaborate.

But another choice, which would maintain CityDesk's benefit of not requiring anything on the server, would be to use a secure peer-to-peer architecture. Something exactly like what Groove provides.

So I thought of porting CityDesk to Groove. Then I noticed that

  1. there is no free Groove runtime. Every single one of my customers would have to buy Groove.
  2. nobody has Groove yet.

That doomed the Groove idea from the start. I talked to some of the Groove "partners" who allegedly are developing software for Groove. "Was the Groove relationship worth anything?" I asked. "HA!" they said. "We paid $1500 and in exchange we get less than 10 clickthroughs a month from their web page. A waste of money. We couldn't even get Groove to share their customer lists with us."

This is not the kind of platform I want to develop for. Yet, technologically, it is exactly the kind of platform I want to develop for, but it's controlled by a greedy (or clueless) company that is going to choke off their own oxygen -- the compelling applications on top of Groove. Ray Ozzie is going nuts about how cool weblogs are -- where's the weblog application for Groove? Who's going to write one? Evan Williams, creator of Blogger? Even Blogger Pro is only $35 a year and that doesn't go very far to paying for a $99 Groove single user license.

What would happen if the Groove runtime were free? Follow the arc of Windows. It started out with a free runtime that let you run one GUI application at a time. Eventually, many people bought the full version to get the benefits of Windows file management, cut and paste between Windows applications, etc. Then Windows 3.0 came out and was so popular and had so many applications that it was bundled with every PC. Today Windows is like the television tax in Britain. Everybody pays it except the developer -- when you're writing software for Windows, it doesn't cost one extra cent. In fact at no time in the history of Windows did developers ever have to worry about the cost of Windows itself.

Anybody who ever tried to sell software components (ActiveX controls, beans, etc.) knows that you have to have a royalty-free runtime or no developer will touch you with a 10 foot pole. Microsoft even lets you redistribute Jet, a complete relational database engine that is 9/10ths of Microsoft Access, for free. Heck, it's preinstalled on Windows 2000.

If Groove wants to be a successful platform, they need to do the same thing. A free Groove redistributable would mean that hundreds of applications would spring up which would get the runtime spread far and wide. Many of those users would see the value in buying a full version of Groove with built in collaboration features. Groove services like the hosted reflectors would have a much larger potential audience.

Of course, they can continue down the path of Notes, assuming that the only way to sell software is to wow CEOs with cool Powerpoints and sell $1,000,000 corporate shelfware licenses. Eventually, this made some real money for Lotus, because Notes had one compelling application -- email -- built in. But imagine if the Notes runtime had been free. If Notes had a software industry sitting on top of it way back in the 80s, some dormroom startup might have made a compelling hypertext application for it instead and preempted the Web. The dreams of huge public Notes networks might have come true. Notes would be as common on PCs as Solitaire. Today it's just Another Email System, one without much of a future.

I keep picking on Groove, but only because there's something interesting in there, one of the few technologies that's interesting enough to care about. Yes, the Groove engineers are architecture astronauts. That's OK. They're building architecture. But they're positioning it like an application and I don't think Groove will be successful if they do that. Someone else will come along with a P2P architecture and sell it like a component, or make it an open source library (yes, I'm aware of JXTA), and that's what software developers will use.


Please Sir May I Have a Linker?

Wednesday, January 28, 2004

For some reason, Microsoft's brilliant and cutting-edge .NET development environment left out one crucial tool... a tool that has been common in software development environments since, oh, about 1950, and taken so much for granted that it's incredibly strange that nobody noticed that .NET doesn't really have one.

The tool in question? A linker. Here's what a linker does. It combines the compiled version of your program with the compiled versions of all the library functions that your program uses. Then, it removes any library functions that your program does not use. Finally, it produces a single executable binary program which people can run on their computers.

Instead, .NET has this idea of a "runtime" ... a big 22 MB steaming heap of code that is linked dynamically and which everybody has to have on their computers before they can use .NET applications.

Runtimes are a problem, much like DLLs, because you can get into trouble when application version 1 was designed to work with runtime version 1, and then runtime version 2 comes out, and suddenly application version 1 doesn't work right for some unpredictable reason. For example right now for some reason our internal company control panel is rounding sales figures to four decimal points, as a result of upgrading from 1.0 to 1.1 of the runtime. Usually the incompatibilities are worse.

In fact .NET includes an extensive technology system called "manifests" which are manifestly complicated and intended to insure that somehow only the right runtime will be used with a given application, but nobody I know can figure out how to use them.

This calls for a story. At the Fog Creek New Year's Eve party, we wanted a bunch of computer screens in the main room to display a countdown until midnight. Michael wrote an application to do this in C# with WinForms in about 60 seconds. It's a great development environment.

My job was getting countdown.exe to run on three computers. Sounds easy.

Nope. Double click the EXE, and I got a ridiculously user-hostile error message about mscoree.dll or something, followed by a gratuitous dump of my path. No mention of the fact that the problem was simply that the .NET runtime was not installed. Luckily I'm a programmer and I figured that must be the problem.

How do you install the runtime? The "easiest" way is through Windows Update. But Windows Update really wanted me to get all the critical updates first before I installed the runtime. That's reasonable, right? Two of the "critical" updates were a Windows service pack and a new version of Internet Explorer, both of which required a reboot.

All told, for each computer I needed to run this little .NET application on, I had to download something like 70 or 80 MB (good thing we have a fast net connection) and reboot three or four times. And this is at a software company! I know how long it took, because the first time it started downloading, I put Office Space on the big screen TV, and by the time the movie was over, the installation process was almost finished. Every ten minutes during the movie I had to jump up, go to each computer, and hit OK to some stupid dialog box.

This is frustrating enough for our in-house apps. But think about our product CityDesk. Almost all of our users download a free trial version before buying the product. The download is around 9 MB and has no additional requirements. Almost none of these users has the .NET runtime yet.

If we asked our trial users, usually small organizations and home users, to go through a movie-length installation hell just to try our app, I think we'd probably lose 95% of them. These are not customers yet, they're prospects, and I can't afford to give up 95% of my prospects just to use a nicer development environment.

"But Joel," people say, "eventually enough people will have the runtime and this problem will go away."

I thought that too, then I realized that every six or twelve months Microsoft ships a new version of the runtime, and the gradually increasing number of people that have it deflates again to zero. And I'll be damned if I'm going to struggle to test my app on three different versions of the runtime just so I can get the benefit of the 1.2% of the installed base that has one of the three.

I just want to link everything I need in a single static EXE that runs without any installation prerequisites. I don't mind if it's a bit bigger. All I would need is the functions that I actually use, the byte code interpreter, and little bit of runtime stuff. I don't need the entire C# compiler which is a part of the runtime. I promise CityDesk doesn't need to compile any C# source code. I don't need all 22 MB. What I need is probably five or six MB, at most.

I know of companies that have the technology to do this, but they can't do it without permission from Microsoft to redistribute bits and pieces of the runtime like the byte code interpreter. So Microsoft, wake up, get us some nice 1950s-era linker technology, and let me make a single EXE that runs on any computer with Win 98 or later and no other external dependencies. Otherwise .NET is fatally flawed for consumer, downloaded software.


Product Vision

Thursday, May 09, 2002

Editor's Note: As I constantly reiterate on these pages, good design requires making often difficult choices between imperfect options. Making those decisions consistently requires a touchstone -- some kind of guiding principle -- often called a vision statement. For my company's product CityDesk , for example, a part of the vision was "content management when you can't run code on your web server." That meant that features which required code on the server, like membership or discussion boards, were left out.

Last week at the Cutter Summit Jim Highsmith described a useful way to figure out your vision statement, which he calls Design the Box. Here's his article, which originally appeared in the 23 August 2001 issue of Cutter Consortium's Agile Project Management E-Mail Advisor and was reprinted with permission. Register here for a free, four-week trial to the E-mail Advisor. - Joel Spolsky

A sample product vision statement:

"For a mid-sized company's marketing and sales departments who need basic CRM functionality, the CRM-Innovator is a Web-based service that provides sales tracking, lead generation, and sales representative support features that improve customer relationships at critical touch points. Unlike other services or package software products, our product provides very capable services at a moderate cost."

This product vision model helps team members pass the elevator test -- the ability to explain the project to someone within two minutes. It comes from Geoffrey Moore's book Crossing the Chasm. It follows the form:

  • For (target customer)
  • Who (statement of the need or opportunity)
  • The (product name) is a (product category)
  • That (key benefit, compelling reason to buy)
  • Unlike (primary competitive alternative)
  • Our product (statement of primary differentiation)

For any project, but particularly those with high uncertainty for which significant requirements changes are anticipated, creating a product vision statement helps teams remain focused on the critical aspects of the product, even when details are changing rapidly. It is very easy to get focused on the short-term issues associated with a 2-4 week development iteration and lose track of the overall product vision.

Even within an IT organization, I think every project should be considered to produce a "product." Whether the project results involve enhancements to an internal accounting system or a new e-commerce site, product-oriented thinking pays back benefits.

One practice that I've found effective in getting teams to think about a product vision is the Design-the-Box exercise (developed originally by colleague Bill Shakelford). This exercise is great to open up a session to initiate a project. In this exercise the entire team, including users, breaks up into groups of four to six (this works best with cross-functional participation). The team makes the assumption that the product will be sold in a shrink-wrapped box, and their task is to design the product box front and back. This involves coming up with a product name, a graphic, three to four key bullet points on the front to "sell" the product, a detailed feature description on the back, and operating requirements.

Coming up with 15 or 20 product features proves to be easy. It's figuring out which 3 or 4 would cause someone to buy the product that is difficult. One thing that usually happens is an intense discussion about who the customers really are. Using a very simple product example, it always amazes me that the product vision boxes end up so disparate among three to five teams. Presentations by each of the groups are then followed by discussing how the multiple focal points can be reduced to a few that everyone agrees upon. A lot of good information gets generated by this exercise -- and it's fun. Once this exercise has been completed, the group can then work on constructing their own version of Moore's product vision statement. Finally, with several hours of active discussion recorded on flip-chart paper, the group can construct a good outline for a complete one- to three-page product vision document that might include: the mission statement, a project profile overview (scope, schedule, cost, defects) with priorities, pictures of the "boxes," target customers and each of their needs, customer satisfaction measures, key technology and operational requirements, critical product constraints (performance, ease of use, volumes), and key financial indicators.

For a four- to six-month project, this visioning exercise might take from a half-day to a full day, but it will pay big dividends. The more critical the delivery schedule and the more volatile the project, the more important it is that the team have a good vision of the final desired outcome. Minus this vision, iterative development projects are likely to become oscillating projects -- going round and round in circles because everyone is looking at the minutia rather than the big picture.

Jim Highsmith is the Practice Director, Agile Project Management, Cutter Consortium .

If you'd like to comment on today's Agile Project Management E-Mail Advisor, send e-mail to apm@cutter.com , or send a letter by fax to +1 781 648 8707 or by mail to The Agile Project Management E-Mail Advisor, Cutter Consortium, 37 Broadway, Suite 1, Arlington, MA 02474-5552, USA.

? Copyright 2001 Cutter Consortium. To sample Cutter's other weekly e-mail advisors, register here.


Project Aardvark Midterm Report

Thursday, July 07, 2005

Project Aardvark, if you've been following along with the blog, is our summer interns' new product. We've got four interns here (three in development, one in marketing) putting together a complete product from beginning to end. Now that they've officially announced what it's all about and we're about to start the first beta, I can bring you up to date on the project, which is more or less halfway done.

The Idea

If you've ever tried to help your technologically-challenged uncle fix his computer problems over the phone, you know what a pain in the butt it can be to try to walk him through the fix.

"Click START"

"What?"

"Start. Click Start. It's in the bottom left."

"I have C - T - R - L in the bottom left."

"The bottom left of your screen."

"Oh. OK, I clicked it."

"OK, now click RUN."

"What?"

"On the menu that came up. Click RUN."

"It's not there."

"What do you mean it's not there?"

"It's not there. I don't have a RUN."

"What do you see? Read me everything you see"

"Recycle Bin... My Computer... Anna Navratilova J P G..."

"No, on the menu."

"What menu?"

"The menu that came up when you clicked start."

"When I what?"

This is when you give up and realize that something that could take you 10 seconds to fix in person is about to become a two hour nightmare during which you'll alienate your family, lose sleep, tie up the phone line while your Auntie Marge is stuck on the turnpike with no gas and can't get through to your uncle to come rescue her, and curse your lot in life. Just because you're a programmer doesn't mean you have to be the help desk for a dozen friends, relatives, and the people in the apartment next door. Does it?

That's the general idea behind the new Fog Creek CopilotSM service. In a nutshell, you go to copilot.com and get an invitation code. You tell your uncle to go to copilot.com and type in that same invitation code. You each get a little program to download and run. When you run the program, your uncle's computer screen shows up in a window. When you move your mouse, his mouse moves. When you type something, it appears on his computer. Etc. And now you fix the problem and log off, and peace is restored and your aunt gets home safely and your uncle dances at your wedding instead of boycotting it and holding up unpleasant signs across from the hotel where said wedding is taking place.

But but but...

Yes, similar services already exist. That never stopped me before. I'd like to point out that Fog Creek has been doubling in revenues every year mostly thanks to bug tracking software, and it's not like we invented bug tracking software. There are a few things our product will do better than the competition, but mostly we just want the Fog Creek Copilot experience to be shockingly seamless. It'll be totally secure, it'll be cheap, it'll be painless, it will work through firewalls on either side so you can help mom at home on her firewalled DSL from behind your NAT at work without a hiccup. We even made it so that the little software program you download is totally self-contained, totally pre-configured, and deletes itself when you're done so you can feel more secure about the whole episode. There's no commitment; you don't have to sign up or create an account and remember a password; you can even make your uncle pay since, after all, he's getting the benefit.

For the geeks in the audience, the service uses a highly customized and optimized version of VNC, but it also requires a customized "reflector" service that we're building which sits outside of any firewalls. The idea is, since you can't connect into mom's computer which is behind a firewall, she'll connect out to our server, you'll connect out to our server too, and the reflector will forward data back and forth between the two of you.

How'd you get the idea?

For the last few years we've been supporting FogBugz customers using a similar scheme, although it's a bit of a pain to set up. Our customers have to follow 7 steps to allow us to control their computers, and we've found that walking people through these steps on the phone takes an average of 5 minutes. With the Fog Creek Copilot service we'll just tell them to check their email and click on a link and hey presto! we're fixing their computer.

So the original idea was to use this for tech support. But when I told the idea to the interns, two out of four said, "yeah, I could use something like that to help my mom." That's when we realized there's a huge world out there of Informal Tech Support... lots of people trying to help Uncle Leo who can't use products like VNC because of the firewall problem. So we changed the focus of release 1 to be the casual and home user instead of tech support departments.

The Name

We had a list of important criteria for the name, but the most important one was that when someone read the name to someone else over the phone, it would be extremely likely that they would get it right. This ruled out names that are weirdly spelled, names which could be easily confused over the phone (for example "m" and "n" are almost impossible to distinguish), and names that have different possible spellings. We went rather too far along the process of investigating the name "Fixant" (complete with a very cool drawing of an ant holding an ethernet cable) before I just got everyone together in a room for a half hour of brainstorming, when we finally hit upon the idea of "Copilot." I can't even remember who thought of it. The idea of brainstorming is just that you shout out ideas, which stimulate other people to have ideas, and you put them all up on a whiteboard.

Well, there are a couple of dozen products named Copilot, many with registered trademarks, so our trademark lawyer advised us to use Fog Creek Copilot which would eliminate any possibility of confusion with those other Copilot brand products. The point of trademark laws is that what you're not allowed to do is create any confusion or potential confusion as to the origin of your product, and sticking "Fog Creek" in front guarantees that, but we have to be religious about always using the full name. I didn't really mind, having started my career working on products like Microsoft Excel, Microsoft Visual Basic for Applications for Microsoft Excel, etc. etc. After a few weeks on the Microsoft Excel team if you ever saw the word "Excel" without a "Microsoft" in front of it, it looked nekkid.

We bought the domain name for more money than we spent developing the first version of FogBugz, oy gevalt, but it is a really good name -- easy to spell, pronounce, and it even sort of suggests what the product does, which makes it more memorable.

The Conference

For some reason, a long long time ago, I had agreed to give the keynote speech for CFUNITED, a conference about ColdFusion.

"I never used ColdFusion!" I protested.

"Don't worry. Nobody has. The biggest sponsor of this conference is Microsoft, who have a huge presence trying to get the ColdFusion developers to switch to VB.NET," the organizers told me.

As luck would have it, the conference timing was perfect for the first feature-complete version of the interns' code. It gave the team a deadline to work to. At the conference itself, we set up a Fog Creek booth and the interns gave demos to several hundred attendees who wandered by.

This was the first trade show Fog Creek had ever attended. The truth is, a trade show is not a very cost-effective way to reach potential customers. Given the cost of travel, hotels, the booth, a thousand bucks for nice brochures, and everybody taking a week off of work, it's a really expensive way to get in front of prospects, especially since I can write an article on my website and get in front of 1000 times as many people.

But that's not really the point: the point is to have interactive experiences with your customers. You can try out lots of different pitches and really listen to how people respond to them, which is something you can't do in non-interactive marketing like web sites and magazine ads. I learned this from Eric Sink, who wrote a great article on the topic, Going to a Trade Show.

We went down to Washington in two big SUVs with all four interns, one of the FogBugz developers, Brett, who gave demos of FogBugz, and me. Our booth probably looked just a little bit too much like a science-fair exhibit, but, whatever, it was our first attempt. Next time we'll know to make backdrop posters that stretch right out to the edge of the backdrop which looks a bit more professional, and I'll remember to bring a lucite brochure dispenser instead of arranging the trifold brochures artfully in the shape of an Aardvark.

But that's not really a big deal. What was a big deal is that we got to talk to hundreds of potential customers, and, wow! the response was just incredible. At the very best, the response we got was, "I need a thousand of these yesterday for my whole team." Almost everyone was impressed by the product and knew that they wanted to use it. A very small number of people were aware of other competitors and other solutions to the problem, but mostly people gave us very positive feedback. More importantly, after spending two days pitching the product again and again to lots of different people we learned the most effective ways to present it. We learned that the best way to present it was not to start with, "You know VNC?" This drew glassy stares. The best way was to start with a typical remote support scenario. "Your mom calls you up. She says her screen is half grey. You have no idea what the heck she's talking about."

The Beta

We got back last Friday and immediately started working on the beta. The goal of this summer internship is to be shipping to paying customers by the end of the summer, and I didn't want our interns to go home after "mostly" finishing the code, leaving us permanent employees to debug until next February, so we're working on a really compressed schedule. Over the next few weeks, we've got to:

  • Deploy the site and service on our web farm
  • Launch a controlled, private beta so a few people can start trying the service and we can figure out if it works in the real world. Yaron is taking beta applications now.
  • Launch a wider public beta
  • Do a round of usability tests in the lab. The nice folks at TechSmith have a product for usability testing called Morae, and they'll be coming out to New York and helping us organize and run the usability tests. Morae lets you set up a complete usability lab using nothing more than their software and a little webcam
  • Start serious QA.

So far we've really been hitting the schedule so I'm pretty confident. In the meantime, you can:

  • Follow along with the Project Aardvark blog!
  • Apply for the beta! If you're accepted, you can try the Fog Creek Copilot service yourself!
  • If you're in New York, come to the Project Aardvark Open House! You'll get free wine and cheese and you can pepper the interns with questions about sockets programming in person. The open house is Thursday, July 14, 2005, from 5:30 PM - 7:00 PM, at Fog Creek Software, 535 8th Ave., 18th Floor, New York.


Reading Code is Like Reading the Talmud

Friday, May 26, 2000

Seth Gordon emailed me some great tips for reading other people's source code:

I started working as a technical writer in Boston after about eight months studying at an Israeli yeshiva.
A year or so ago, I and a co-worker were trying to decipher a yacc-and-Perl monstrosity that generated some of our documentation. We sat down with printouts of the code and went through it, virtually line by line. "Now, because of the way yacc works, when it encounters *this* kind of data, it will run *this* function ... what does *this* line do? ... at *this* point, what does that messy data structure look like?..."
Another co-worker asked me how it was going, and I told him, "It's just like reading Talmud."
The following Talmud-reading tactics are, I think, also useful for code-reading:
  1. Work in pairs, thinking out loud to one another.
  2. Argue. If your partner says "this means X", and you either don't understand why or you have another opinion, demand an explanation.
  3. Sometimes, when dealing with a chunk of text, it's easier to figure out the middle *after* you understand what's on both ends. Therefore, if a fragment of text has you stumped, try skipping over it and seeing if you can come back to it later. (But you still have to come back to it eventually.)
  4. Read the text both "inside" and "outside". An inside reading translates the text into English (or whatever your native language is) phrase-by-phrase; an outside reading translates a larger chunk into an idiomatic paragraph. If you only read inside, you can miss the forest for the trees; if you only read outside, you can fool yourself by making broad guesses and not verifying them with details.
Thanks, Seth! Great ideas!


REALBasic

Monday, June 12, 2000

REALBasic comes tantalizingly close to solving a real problem; with a few minor changes, it could become the next great development tool for GUI programming. But it's not there yet.

Here's my view on development tools for creating commercial software that needs to run on Windows and/or Mac (i.e., 99% of the desktop). There are some important axioms.

Axiom one. The universe of programming languages is more or less divided into two categories: programming languages with memory management built in (garbage collection, like Java, or reference counting, like Basic), and programming languages that make the programmer do the memory management (C/C++). Programmers are much more productive in languages that do the memory management for them. Over the last two years I spent a lot of time writing GUI code in C++ (using MFC) and a lot of time writing GUI code in Visual Basic. Know what? My time writing Visual Basic was ALWAYS significantly more productive. Other programmers who have used both environments extensively have always agreed with me on this point. Programmers who don't agree with me have not used both. There are a lot of C++ only programmers who think of Basic as a toy language and disdain it. They think that they are cool coding jocks because they don't need wimpy languages. Frankly, they are wrong, and I don't have very much patience for people who get religious about tools and judge things that they haven't used.

Memory management languages (Basic, Java) tend to be slower. This may not matter in the long run. However, as of now, there are still lots of examples where you can speed up code several thousand fold by writing some tight C++ to replace Basic code. As an example, I once wrote some code in Visual Basic to convert the contents of a rich edit box to HTML. It took about 10 seconds for a page of rich text. When I converted it to C++, I was able to use much lower level messages to the underlying rich edit control, and the speedup was profound: instead of 10 seconds, it took a few milliseconds.

But for most GUI tasks, which operate at "user speed" (most of the time is spent waiting for the next message from the user), there is no benefit to using C++. Given axiom one, my preferred development environment would usually be something like Visual Basic.

Axiom two. Users have expectations for what makes up a "professional looking" application. If you violate these expectations, they will feel like your program is a piece of junk. For example, on the Mac, they expect the menus to be at the top of the screen. And they expect that while they are editing text, the whole screen won't flash, which means you absolutely have to use double-buffering. They usually expect you to use "native widgets". When programmers try to reinvent their own widgets (viz. Netscape 6), users will think the program looks clunky. If you use the JDK, and your Windows application uses the Java coffee cup as its icon, users will think that your program is not professional. People are less likely to buy a product that looks unprofessional. I don't know how many people have told me that they tried various applications which were written in Java and thought that they felt unprofessional. They were slow, they flashed a lot, their widgets behaved slightly wrong. Thus, there is a lot of value in giving software developers absolute control over the exact look and feel.

When you add in axiom two, and the need for speed in certain critical areas, I would modify my statement a bit, and say that my preferred development environment would be Visual Basic, with the ability to drop into C++ to do certain things. VB gives you at least two good ways to do this: through DLLs or through COM objects. COM objects that are also controls even give you the ability to define new user interfaces.

Now let's talk about platforms. Windows has 90% or 95% of the desktop market. Macintosh has 5% or 10%. If you're talking about office users, Windows is even more dominant; Macintosh has a bit more of the home users, but probably not even near 15%. What this means is that if you are a software developer, the only thing that makes sense financially is to develop a Windows version first. Then, you need to evaluate the cost of doing a Mac version. If that cost is only 10% more, it's worth it. If that cost is something like 50% more, it's not worth it. If I have a product that cost me $1,000,000 to develop, and 10,000 Windows users are using it, that's $100 per user. Now if I have to make a Mac version, and it's going to cost me $500,000 to port the Windows version, and the product is going to be just as popular among Mac users as Windows users, then I will have about 1000 Mac users. That means that my port cost me $500 per user.

This is not a good proposition. I'd rather spend the money getting more Windows users, because they're cheaper.

What does this mean to you, a Mac tools developer? It means that your number one priority is making it possible for Windows developers to port to the Mac for less than 10% of the original cost. The big companies that have products on Mac and Windows (Macromedia, Quark, Adobe, etc) generally started on the Mac. That means that it was worth almost any amount of money to port their product to Windows, because the market was so much larger. The only big exception is Microsoft, which has several different portability layers that allow them to develop one product for both platforms. For products at Microsoft which already have portability layers (Project, Word, Excel), they can do a Mac version for less than 10% extra, so they do it. For other products (Access, FrontPage), the cost of a Mac port would be more than 10%, so they don't do it.

An interesting point is that today, try as one might, there is just no great way to create a Mac port for less than 10% of the cost of the Windows original. Which is why not much stuff shows up on the Mac anymore, unless it's strategic in some way (like the Real Media Player).

In my view, the "holy grail" of programming would be an environment which:

  • allowed me to develop GUIs in a high productivity GUI environment like Visual Basic
  • allowed me to drop into C++ and create COM objects for things which need to be fast or have a very particular look and feel

So far, this environment exists on Windows (VB with VC++), and almost on the Mac (REALBasic with CodeWarrior, although they are not glued together nicely with an architecture like COM Controls).

Now, if you want me to make a Mac port, I need the third thing:

  • cost of porting from Windows to Mac is less than 10%.

Just because of the differences in fonts between Mac and Windows, I'm already going to have to lay out every dialog again (and swap the position of all the OK and Cancel buttons) and change words like "Directory" to "Folder". So that uses up, like, 8%, which means that porting the code itself better be super-duper easy, like, 2% of the effort.

And REALBasic can't do that today. But it is soooooooooo close, it's painful. I want to be able to write code in VB on Windows, and have it just work in REALBasic. No complicated one-way conversion scripts, because when I'm coding and porting, I'm coding all day long. There CAN'T BE a conversion step, because if I find a bug in the original version, I can't start again with the port.

I think that the number one priority for REALBasic would be to make it compile Visual Basic code directly. There are too many cases where REALBasic is gratuitously different from Visual Basic. As a small example, it lacks a SET statement, instead using LET to do SET, which removes the possibility of using default properties.

There are functions which have been in Basic since time immemorial, like LCase, which REALBasic calls "LowerCase" for absolutely no good reason... except to serve as an obstacle to porting to or from VB.

It needs to be able to do COM controls. I need to be able to set up an environment where I can code once for Windows, using VB and COM controls written in VC++, and have that code compile unchanged with REAL and CodeWarrior.

There are some cases where REALBasic includes a "neat" extra feature which somebody thought was cool, like "//" comments. These aren't bad features, but they need to be turned off by default. That way, I can be sure that if I write my code in REALBasic, it's going to work in Visual Basic later. Somebody needs to go over VB and REALBasic with a fine tooth comb, find every small difference, and fix them! The amount of work this will take is very small, which is why I said that REALBasic is sooooooooooo close.

There are other useful things besides pure compatibility: tools that take into account the fact that my code needs to run on both platforms. For example, since every dialog will have to be laid out twice, REAL Basic should have a tool that lets me manage two versions of the layout of each form automatically. It should detect when one version has changed and remind me somewhere that I need to change it for the other platform. It should have good #ifdef capability which is 100% compatible with VB. It needs to provide small compatibility libraries, for example, for parsing file paths, that I can use in VB for Windows as well, so that I have a chance of writing portable code.

And the prospective gains are HUGE. Visual Basic is the single best selling programming language in history, by a large margin. If REALBasic can get that cost of porting below 10%, a flood of developers will suddenly discover that porting to the Mac is worthwhile, which will hugely improve the sales of REALBasic. In the long run, it will also be shockingly beneficial to Apple, which has its own chicken and egg problem trying to get software for the Macintosh. If Apple has any sense at all (and they don't), they would PAY REALBasic to be more VB compatible.


Rick Chapman is In Search of Stupidity

Friday, August 01, 2003

(This is the foreword to Rick Chapman's new book, In Search of Stupidity .)

In every high tech company I?euro;™ve known, there?euro;™s a war going on, between the geeks and the suits.

Before you start reading this great new book full of propaganda from software marketing wizard and über-suit Rick Chapman, let me take a moment to tell you what the geeks think.

Play along with me for a minute, will you?

Please imagine the most stereotypically pale, Jolt-drinking, Chinese-food-eating, video-game-playing, slashdot-reading Linux-command-line-dwelling dork. Since this is just a stereotype, you should be free to imagine either a runt or a kind of chubby fellow, but in either case this is not the kind of person who plays football with his high school pals when he visits mom for Thanksgiving. Also, since he?euro;™s a stereotype, I shall not have to make complicated excuses for making him a him.

This is what our stereotypical programmer thinks: ?euro;œMicrosoft makes inferior products, but they have superior marketing, so everybody buys their stuff.?euro;?

Ask him what he thinks about the marketing people in his own company. ?euro;œThey?euro;™re really stupid. Yesterday I got into a big argument with this stupid sales chick in the break room and after ten minutes it was totally clear that she had no clue what the difference between 802.11a and 802.11b is. Duh!?euro;?

What do marketing people do, young geek? ?euro;œI don?euro;™t know. They play golf with customers or something, when they?euro;™re not making me correct their idiot spec sheets. If it was up to me I?euro;™d fire ?euro;˜em all.?euro;?

A nice fellow named Jeffrey Tarter used to publish an annual list of the hundred largest personal computer software publishers called the Soft-letter 100. Here?euro;™s what the top ten looked like in 1984[1]:


Rub a dub dub

Wednesday, January 23, 2002

One reason people are tempted to rewrite their entire code base from scratch is that the original code base wasn't designed for what it's doing. It was designed as a prototype, an experiment, a learning exercise, a way to go from zero to IPO in nine months, or a one-off demo. And now it has grown into a big mess that's fragile and impossible to add code to, and everybody's whiny, and the old programmers quit in despair and the new ones that are brought in can't make head or tail of the code so they somehow convince management to give up and start over while Microsoft takes over their business. Today let me tell you a story about what they could have done instead.

FogBUGZ started out six years ago as an experiment to teach myself ASP programming. Pretty soon it became an in-house bug tracking system. It got embellished almost daily with features that people needed until it was good enough that it no longer justified any more work.

Various friends asked me if they could use FogBUGZ at their companies. The trouble was, it had too many hardcoded things that made it a pain to run anywhere other than on the original computer where it was deployed. I had used a bunch of SQL Server stored procedures, which meant that you needed SQL Server to run FogBUGZ, which was expensive and overkill for some of the two person teams that wanted to run it. And so on. So I would tell my friends, "gosh, for $5000 in consulting fees, I'll spend a couple of days and clean up the code so you can run it on your server using Access instead of SQL Server." Generally, my friends thought this was too expensive.

After this happened a few times I had a revelation -- if I could sell the same program to, say, three people, I could charge $2000 and come out ahead. Or thirty people for $200. Software is neat like that. So in late 2000, Michael sat down, ported the code so that it worked on Access or SQL Server, pulled all the site-specific stuff out into a header file, and we started selling the thing. I didn't really expect much to come of it.

In those days, I thought, golly, there are zillions of bug tracking packages out there. Every programmer has written a dinky bug tracking package. Why would anyone buy ours? I knew one thing: programmers who start businesses often have the bad habit of thinking everybody else is a programmer just like them and wants the same stuff as them, and so they have an unhealthy tendency to start businesses that sell programming tools. That's why you see so many scrawny companies hawking source-code-generating geegaws, error catching and emailing geegaws, debugging geegaws, syntax-coloring editing tchotchkes, ftping baubles, and, ahem, bug tracking packages. All kinds of stuff that only a programmer could love. I had no intention of falling into that trap!

Of course, nothing ever works out exactly as planned. FogBUGZ was popular. Really popular. It accounts for a significant chunk of Fog Creek's revenue and sales are growing steadily. The People won't stop buying it.

So we did version 2.0. This was an attempt to add some of the most obviously needed features. While David worked on version 2.0 we honestly didn't think it was worth that much effort, so he tended to do things in what you might call an "expedient" fashion rather than, say, an "elegant" fashion. Certain, ahem, design issues in the original code were allowed to fester. There were two complete sets of nearly identical code for drawing the main bug-editing page. SQL statements were scattered throughout the HTML hither and yon, to and fro, pho and ton. Our HTML was creaky and designed for those ancient browsers that were so buggy they could crash loading about:blank.

Yeah, it worked brilliantly, we've been at zero known bugs for a while now. But inside, the code was, to use the technical term, a "big mess." Adding new features was a hemorrhoid. To add one field to the central bug table would probably require 50 modifications, and you'd still be finding places you forgot to modify long after you bought your first family carplane for those weekend trips to your beach house on Mars.

A lesser company, perhaps one run by an executive from the express-package-delivery business, might have decided to scrap the code and start over.

Did I mention that I don't believe in starting from scratch? I guess I talk about that a lot.

Anyway, instead of starting from scratch, I decided it was worth three weeks of my life to completely scrub the code. Rub a dub dub. In the spirit of refactoring, I set out a few rules for this exercise.

  1. No New Features, not even small ones, would be added.
  2. At any time, with any check in, the code would still work perfectly.
  3. All I would do is logical transformations -- the kinds of things that are almost mechanical and which you can convince yourself immediately will not change the behavior of the code.

I went through each source file, one at a time, top to bottom, looking at code, thinking about how it could be better structured, and making simple changes. Here are some of the kinds of things I did during these three weeks:

  • Changed all HTML to xhtml. For example, <BR> became <br />, all attributes were quoted, all nested tags were matched up, and all pages were validated.
  • Removed all formatting (<FONT> tags etc.) and put everything in a CSS style sheet.
  • Removed all SQL statements from the presentation code and indeed all program logic (what the marketing types like to call "business rules"). This stuff went into classes which were not really designed -- I simply added methods lazily as I discovered a need for them. (Somewhere, someone with a big stack of 4x6 cards is sharpening their pencil to poke my eyes out. What do you mean you didn't design your classes?)
  • Found repeated blocks of code and created classes, functions, or methods to eliminate the repetition. Broke up large functions into multiple smaller ones.
  • Removed all remaining English language text from the main code and isolated that in a single file for easy internationalization.
  • Restructured the ASP site so there is a single entry point instead of lots of different files. This makes it very easy to do things that used to be hard, for example, now we can display input error messages in the very form where the invalid input occurred, something which should be easy if you lay things out right, but I hadn't laid things out right when I was learning ASP programming way back when.

Over three weeks the code got better and better internally. To the end user, not too much changed. Some fonts are a little nicer thanks to CSS. I could have stopped at any point, because at any given time I had 100% working code (and I uploaded every checkin to our live internal FogBUGZ server to make sure). And in fact I never really had to think very hard, and I never had to design a thing, because all I was doing was simple, logical transformations. Occassionally I would encounter a weird nugget in the code. These nuggets were usually bug fixes that had been implemented over the years. Luckily I could keep the bug fix intact. In many of these cases, I realized that had I started from scratch, I would have made the same bug all over again, and may not have noticed it for months or years.

I'm basically done now. It took, as planned, three weeks. Almost every line of code is different now. Yep, I looked at every line of code, and changed most of them. The structure is completely different. All the bug-tracking functionality is completely separate from the HTML UI functionality.

Here are all the good things about my code scrubbing activity:

  • It took vastly less time than a complete rewrite. Let's assume (based on how long it took us to get this far with FogBUGZ) that a complete rewrite would have taken a year. Well, that means I saved 49 weeks of work. Those 49 weeks represent knowledge in the design of the code that I preserved intact. I never had to think, "oh, I need a new line here." I just had to change <BR> to <br /> mindlessly and move on. I didn't have to figure out how to get multipart working for file uploads. It works. Just tidy it up a bit.
  • I didn't introduce any new bugs. A couple of tiny ones, of course, probably got through. But I was never doing the types of things that cause bugs.
  • I could have stopped and shipped at any time if necessary.
  • The schedule was entirely predictable. After a week of this, you can calculate exactly how many lines of code you clean in an hour, and get a darn good estimate for the rest of the project. Try that, Mozilla river-drawing people.
  • The code is now much more amenable to new features. We'll probably earn back the three weeks with the first new major feature we implement.

Much of the literature on refactoring is attributable to Martin Fowler, although of course the principles of code cleanups have been well known to programmers for years. An interesting new area is refactoring tools, which is just a fancy word for programs that do some of this stuff automatically. We're a long way from having all the good tools we need -- in most programming environments you can't even do a simple transformation like changing the name of a variable (and having all the references to it change automatically). But it's getting better, and if you want to start one of those scrawny companies hawking programming tool geegaws, or make a useful contribution to open source, the field is wide open.


Set Your Priorities

Wednesday, October 12, 2005

It was getting time to stop futzing around with FogBugz 4.0 and start working on 5.0. We just shipped a big service pack, fixing a zillion tiny little bugs that nobody would ever come across (and introducing a couple of new tiny little bugs that nobody will ever come across) and it was time to start adding some gen-yoo-ine new features.

By the time we were ready to start development, we had enough ideas for improvement to occupy 1700 programmers for a few decades. Unfortunately, all we have is three programmers, and we wanted to be shipping next fall, so there had to be some prioritization.

Before I tell you how we prioritized our list of features, let me tell you two ways not to do it.

Number one. If you ever find yourself implementing a feature simply because it has been promised to one customer, RED DANGER LIGHTS should be going off in your head. If you're doing things for one customer, you've either got a loose cannon sales person, or you're slipping dangerously down the slope towards consultingware. And there's nothing wrong with consultingware; it's a very comfortable slope to slip down, but it's just not as profitable as shrinkwrap software.

Shrinkwrap is the take-it-or-leave it model of software development. You develop software, wrap it in plastic, and customers either buy it, or they don't. They don't offer to buy it if you implement just one more feature. They don't call you up and negotiate features. You can't call up Microsoft and tell them, "Hey, I love that BAHTTEXT function you have in Excel for spelling out numbers in Thai, but I could really use an equivalent function for English. I'll buy Excel if you implement that function." Because if you did call up Microsoft here is what they would say to you:

"Thank you for calling Microsoft. If you are calling with a designated 4-digit advertisement code, press 1. For technical support on all Microsoft products, press 2. For Microsoft presales product licensing or program information, press 3. If you know the person at Microsoft you wish to speak to, press 4. To repeat, press Star."

Notice? None of the choices was, "To negotiate what features need to be added to our products before you'll buy them, press 5."

Custom development is that murky world where a customer tells you what to build, and you say, "are you sure?" and they say yes, and you make an absolutely beautiful spec, and say, "is this what you want?" and they say yes, and you make them sign the spec in indelible ink, nay, blood, and they do, and then you build that thing they signed off on, promptly, precisely and exactly, and they see it and they are horrified and shocked, and you spend the rest of the week reading up on whether your E&O insurance is going to cover the legal fees for the lawsuit you've gotten yourself into or merely the settlement cost. Or, if you're really lucky, the customer will smile wanly and put your code in a drawer and never use it again and never call you back.

Somewhere in the middle is consultingware, where you pretend to be doing shrinkwrap while really doing custom development. Here's how consultingware works:

  1. You're working as a wage-slave writing code for a shoe company, and
  2. the company needs shoe-shining software, so
  3. you develop shoe-shining software in VB 3.0 using bits and pieces of JavaScript, Franz Lisp, and a FileMaker database running on an old Mac connected over the network using AppleScript, then
  4. everyone thinks it's the cat's whiskers, so, always having dreamed of starting your own software company and maybe being Bill Gates or perhaps even just Larry Ellison
  5. you buy the rights to ShoeShiner 1.0 from your old company and get VC to start your own company, ShoeShiner LLC, marketing shoe-shining software, but
  6. none of your beta testers can get it to work because of the obscure dependencies on AppleScript and the hardcoded IP addresses in the source code, so it takes a month to install at each client site, and
  7. you have trouble getting clients, because your product is so expensive because of all the installation costs, including the vintage Macintosh IIci running System 7 which they have to buy on ebay from computer museums, so your VCs start to get really nervous,
  8. putting pressure on the sales guys,
  9. who find out that one of your potential customers doesn't need a shoe-shining thing but he could really use trousers-pressing software, and
  10. the sales guy, being a sales guy, sells him $100K worth of trousers-pressing software,
  11. and now you spend 6 months writing a one-off "trousers-pressing module" for this client, which
  12. no other client will ever need, thus, effectively,
  13. for all intents and purposes you've just spent a year raising VC so that you could work as a wage-slave writing code for a trouser company; GOTO 1.

Sparky, I'm gonna have to strongly recommend clinging as strongly as possible to the shrinkwrap side of the equation. That's because shrinkwrap has no marginal costs for each additional customer, so you can essentially sell the same thing over and over again and make a lot more profit. Not only that, but you can lower the price, because you can spread your development costs out over a lot more customers, and lowering the price gets you more customers because more people will suddenly find your now-cheaper software worthwhile, and life is good and all is sweet.

Thus. If you ever find yourself implementing a feature simply because it has been promised to a customer, you're drifting over to the land of consultingware and custom development, which is a fine world to operate in if that's what you like, but it just doesn't have the profit potential of off-the-shelf commercial software.

Now, I'm not saying you shouldn't listen to your customers. I for one think that it's about time Microsoft actually implemented a version of the BAHTTEXT function for those of us who haven't yet joined the global economy and learned Thai and who still write checks using other currencies. And in fact if you want to tell yourself that the best way to allocate your development resources is effectively to let your biggest customers "bid" on features, well, you can do that too, although you'll soon find that the kind of features that big, rich customers want are not the same as the kind of features that the mass market wants, and that feature you put in to handle Baht currency is not really helping you sell Excel to health spas in Scottsdale, Arizona, and in fact what you're really doing is letting your sales force pimp out your developers with the sole goal of maximizing their personal commissions.

The path to being Bill Gates, this is not.

Now, let me tell you the second way not to decide what features to implement. Don't do things just because they're inevitable. Inevitability is not a high enough bar. Let me explain.

Some time during the first year of Fog Creek's operations, I was filing away some papers, and realized that I was all out of blue folders.

Now, I have a system. Blue folders are for clients. Manila folders are for employees. Red folders are receipts. Everything else is yellow. I needed a blue folder and had run out.

So I said to myself, "What the heck, I'm going to need a blue folder eventually anyway, I might as well go to Staples and buy some now."

Which was, of course, a waste of time.

In fact when I thought about this later, I realized that for a long time, I had been doing dumb shit (that's a technical term) simply because I figured that eventually it would have to get done, so I might as well do it now.

I used this excuse to weed the garden, patch holes in the walls, sort out MSDN disks (by color, language, and number), etc., etc., when I should have been writing code or selling code, the only two things a startup really needs to do.

In other words, I found myself pretending that all non-optional tasks were equally important, and therefore, since they were inevitable anyway, they could be done in any order! Tada!

But to be honest, I was just procrastinating.

What should I have done? Well, for starters, I could get over my fetish for having file folders all be the right color. It just doesn't make any difference. You don't have to color-code your files.

Oh, and those MSDN CD-ROMs? Toss them in a big box. PER-fect.

More importantly, I should have realized that "important" is not a binary thing, it's an analog thing. There are all kinds of different shades of important, and if you try to do everything, you'll never get anything done.

So if you want to get things done, you positively have to understand at any given point in time what is the most important thing to get done right now and if you're not doing it, you're not making progress at the fastest possible rate.

Slowly, I'm weaning myself off of my tendency to procrastinate. I'm doing this by letting less-important things go undone. There's some nice lady from the insurance company who has been pestering me for two months to get some data she needs to renew our policy, and I didn't actually get her the data until she asked about the fiftieth time, along with a stern warning that our insurance is going to expire in three days. And this is a good thing, I think. I've grown to think that keeping your desk clean is actually probably a sign that you're not being effective.

How's that for a mortifying thought!

So. Don't do features based on what the sales force has inadvertently promised a single customer, and don't do unimportant-slash-fun features first because "you're going to have to do them eventually anyway."

Anyway, back on the topic of choosing features for FogBugz 5.0. Here's how we got our initial prioritization.

First, I took a stack of 5x8 cards, and wrote a feature on each one. Then I called the team together. In my experience this works with up to about twenty people, and it's a good idea to get as many different perspectives in the room: programmers, designers, people who talk to customers, sales, management, documentation writers and testers, even (!) customers.

I asked everyone to bring their own list of feature ideas to the meeting, too. The first part of the meeting was going over each feature very, very quickly and making sure we had a very, very rough common understanding of what the feature was, and that each feature had a card.

At this stage, the idea was not to debate any feature on its merits, or to design the feature, or even to discuss the feature: just to have a vague, rough idea of what it was. Some of the features for FogBugz 5.0 were things like

  • Personalizable Home Page
  • Painless Software Schedules
  • Track Billable Time
  • Fork a bug
  • (46 others...)

Very vague stuff. Remember we didn't need to know at this point how each feature would be implemented, or what it involved, because our only goal was getting a rough prioritization that could be used as the basis to start development. This got us a list of about 50 big features.

In part two, we went through all of the features and everybody voted on each feature: just a quick "thumbs up" or "thumbs down." No discussion, no nothing: just a real quick thumbs up or thumbs down on each feature. This revealed that about 14 of the feature ideas didn't have much support. I threw out all the features that only got one or two votes, leaving us with 36 potential features.

Next we assigned costs for each of these features, on a scale of 1 to 10, where 1 was a quicky feature and 10 was a big monster feature. Here it's important to remember that the goal was not to schedule the features: just to separate the tiny features from the medium features from the huge features. I just went through each of the features and asked the developers to call out "small," "medium," or "large." Even without knowing how long a feature is going to take, it's easy to see that forking a bug is a "small" feature while the big, vague "Personalizable Home Page" feature was large. Based on the consensus estimate of costs and my own judgment, we put down prices on all the features:

Once again, it's really messy, it's not exact, and it doesn't matter. You're not making a schedule today: you're just prioritizing. The only thing that you have to get approximately right is the vague idea that you could do two medium features or one large feature or ten small features in about the same amount of time. It doesn't have to be accurate.

The next step was making a menu of all thirty proposed features and their "costs". Everybody on the team got a copy of the menu and was given $50 to play with. They could allocate their money any way they wanted, but they only had $50 to spend. They could buy half-features, if they wanted, or buy double features. Someone who really liked that Track Billable Time feature could spend $10 or $15 on it; someone who liked it a little might only spend $1 and hope that enough other people funded it.

Next, we added up how much everyone spent on each feature:

Finally I divided the amount spent by the cost:

And then sorted by this number to find the most popular features:

Ta da! A list of all the features you might want to do, in rough order of everyone's best idea of which features are the most important.

And now you can start to refine. For example, you can clump together features that naturally belong together, for example, doing software schedules makes billable time easier, so maybe we should either do both or neither. And sometimes looking down the prioritized list it's just real obvious that something is messed up. So, you change it! Nothing is carved in stone. You can even change the prioritization as you go through development.

But what surprised me the most is that the final list we produced was really a very good prioritization for FogBugz 5.0, and really did reflect our collective consensus about the relative priorities of various features.

Priority list in hand, we set out to more or less work down the list in order until about March, when we plan to stop adding new features and start the integration and testing phase. We'll write specs for each (nonobvious) feature right before implementing that feature.

(The nattering scorekeepers of the BDUF/Agile beauty contest are now thoroughly confused. "Was that a vote for BDUF? Or Agile? What does he want? Can't he just take sides for once?!")

The whole planning process took three hours.

If you're lucky enough to have the ability to release software more frequently than we do, (see Picking a Ship Date), you still need to work down the list in order, but you can just stop and do releases more often. The good thing about frequent releases is that you can reprioritize the list regularly based on actual customer feedback, but not every product has this luxury.

Mike Conte taught me this system during the planning of Excel 5, where it only took a couple of hours even with a couple of dozen people in a conference room. The cool thing was that the roughly 50% of the features that we didn't have time to do were really stupid features, and Excel was better because it didn't have them.

It's not perfect, but it's better than going to Staples to get blue folders, I'll tell ya that.


Spring in Cambridge

Monday, March 19, 2001

Even though the weather is in the 40s, and the wind chill is wintry, today I woke up and went outside and felt like spring was finally in the air. Maybe I started to notice this last Friday when I went back to the Fog Creek garden in a t-shirt and wasn't actually cold.

Spring conjures up a funny set of associations for me. I think about the fact that it's still daylight when you go outside after dinner. I think about living in a nice old messy house with real wood floors, lots and lots of books, and flowers in the garden. I think about two of my favorite places in the spring, Berkeley and Boston, cities that revolve around learning and education and the unbridled enthusiasm of youth and the belief that anything is possible. As a first year student I arrived at university without even the basic premise of the shape my life would take, but compared to the unspeakably horrible time I had doing compulsory military duty, any shape of life sounded utopian to me. I could be in theatre! I could write for the school newspaper! Politics! Hacking! Teaching! Art! I could become a competitive swimmer! Play piano! All possible!

Around 1993, I think, I finally got access to the World Wide Web at home. $35 a month for SLIP from Panix. One of the first things that captivated me was Travels with Samantha, by Philip Greenspun. The pictures were almost impossible to make out in 16 VGA colors; the connection was at 14.4 baud; the screen was 640x480. But Greenspun's style of personal journal was captivating. He told us how lonely it was to live in Las Alamos for three months. He told us how to build an online community. He told us about his $5000 Viking stove.

Greenspun has been a key inspiration to many of us. He started a company called arsDigita, which was really just another consulting company for the Internet, but it had two things which made it uniquely different.

It had a personal voice. When you went to the home page, there was a happy little note about the new offices or the latest course offerings; the style was not boring corporate/marcomm happy-talk, it was Philip telling you that if you're poor, there are some free services, but if you're rich, head on over here and we'll build you a nice service of your own, but bring a lot of money please, because the future's so bright you gotta wear shades.

And it had education. The web was new and exciting and offered unlimited possibilities. Everybody wanted to learn about it and arsDigita would teach you, for free, starting with a book that cost about $15 more than it should because it was stuffed with completely irrelevant full-color photographs that seemed to be there solely to assuage the author's egotistical photography hobby, but which really were so bright and colorful and optimistic that nobody could read that book (or any of the photography-filled pages of Greenspun's websites) without becoming optimistic and excited about the future of the Internet, and when you did, there was a whole chapter on how to start a consulting company just like Philip and get rich off of the Internet, complete with suggested prices and detailed calculations of how you will make $250,000 a year and be able to buy $5000 Viking stoves and a beautiful condo in Harvard Square, in the spring, with flowers budding and optimist young students everywhere practically popping out of their tanktops. Stream of consciousness, random, and interspersed with completely random bogus snippets of half-baked Oracle SQL statements but goddamn it, there was a personal voice there.

(I know, anybody at arsDigita will tell you "no, Joel! It's open source that made arsDigita different." Or community. Or content management software. Sorry, all that stuff bores me. What excites me was the way arsDigita had a personal voice on the Internet that made it possible to relate to it in a human way, which is what people want to do, since we're humans.)

Fog Creek Software was inspired in no small part by arsDigita; the code name for the company was "PaxDigita" and we took inspiration from Greenspun's tongue-in-cheek corporate slogan "We don't have venture capital; we have money." ArsDigita thought that it was enough to be profitable.

There were two things I thought that arsDigita did wrong. First of all, they didn't understand that consulting doesn't scale so well; if you really want to make an impact, you need to license software. A consultant working on a project makes ten or twenty percent profit. If you license software, each marginal unit you sell is 100% profit. That's why software companies have valuations that are something like 25 times richer than consulting companies. Consulting is important and keeps us in touch with customers, and it funds the software development, but our goal is to be a software company. (ArsDigita seems to have learned this lesson. Allen Shaheen, the CEO, writes that "we are also considering developing and marketing additional software products using a different type of licensing arrangement. This investment is in addition to our investment in ACS as an open source product. We are considering this approach because it would allow us to accelerate our development of new and expanded features." That's happy talk for "the new features are going to cost you from now on.")

The other thing I didn't like about arsDigita was an almost religious aversion to Microsoft. There was a deep belief that arsDigita "eliminated risk" by never using a Microsoft product. The truth is, they just didn't know anything about Microsoft products. Their religious bigotry struck me as just that: bigotry. In the words of Lazarus Long, "I don't believe anything. Gets in the way of learning." If you say, "we know Unix better, so we get better results with Unix," you're a scientist. If you say, "we don't use any Microsoft crap, it all sucks" you're just a fanatic, and it's getting in the way of doing good work. (Greenspun has learned this lesson too; almost all of his bitterness to Microsoft seems to have dissipated and he finally admits, "I'm not motivated to fight a religious war for any particular database management system or Web server." It's nice to watch smart people learning.)

But there's one thing that arsDigita did very, very, right, and that was the personal voice thing. The biggest thing I think about with Fog Creek is how to maintain that personal voice, and if we can do it, I'll owe a big, big debt to Philip Greenspun.

For ArsDigita, the spring is over. ArsDigita found it impossible to hire engineers during the dotcom bubble without the promises of IPO riches, and took $38 million in venture capital. They grew from 5 to 250 people.

And then the excited exuberance gave way to a completely collapsed market for Internet consulting services.

The VCs promptly installed boring old-school management who insisted on a happy-talk marcomm website with complete bullshit like "Fully integrated through a single data model to ensure consistency across all business processes, ACS empowers organizations to create Web services that maximize value for the end-user."

Philip left.

The voice is gone.

To me, arsDigita will always remind me of those silly irrelevant pictures of pretty girls rollerblading and clam stands and boats in the Charles and sunsets in New Mexico and flowers and spring, and if you built it, they will come, and if you have a voice on your website, they will care. When I go to the website today and read that they "maximize value for the end-user," I think of that dazzling, exciting kid you knew freshman year, who jumped from varsity squash to journalism to drama to campus politics, and was all fired up to change the world. But he got married, has 2 1/2 kids, took a job as an insurance agent and wears a gray suit to work every day.

Maturity is kind of sad.

But spring is in the air, and that makes up for it.


Strategy Letter I: Ben and Jerry's vs. Amazon

Friday, May 12, 2000

Building a company? You've got one very important decision to make, because it affects everything else you do. No matter what else you do, you absolutely must figure out which camp you're in, and gear everything you do accordingly, or you're going to have a disaster on your hands.

The decision? Whether to grow slowly, organically, and profitably, or whether to have a big bang with very fast growth and lots of capital.

The organic model is to start small, with limited goals, and slowly build a business over a long period of time. I'm going to call this the Ben and Jerry's model, because Ben and Jerry's fits this model pretty well.

The other model, popularly called "Get Big Fast" (a.k.a. "Land Grab"), requires you to raise a lot of capital, and work as quickly as possible to get big fast without concern for profitability. I'm going to call this the Amazon model, because Jeff Bezos, the founder of Amazon, has practically become the celebrity spokesmodel for Get Big Fast.

Let's look at some of the differences between these models. The first thing to ask is: are you going into a business that has competition, or not?


Strategy Letter II: Chicken and Egg Problems

Wednesday, May 24, 2000

The idea of advertising is to lie without getting caught. Most companies, when they run an advertising campaign, simply take the most unfortunate truth about their company, turn it upside down ("lie"), and drill that lie home. Let's call it "proof by repeated assertion." For example, plane travel is cramped and uncomfortable and airline employees are rude and unpleasant, indeed the whole commercial air system is designed as a means of torture. So almost all airline ads are going to be about how comfortable and pleasant it is to fly and how pampered you will be every step of the way. When British airways showed an ad with a businessman in a plane seat dreaming that he was a baby in a basket, all sense of reasonableness was gone for good.

Need another example? Paper companies are completely devastating our national forests, clear cutting old growth forest which they don't even own. So when they advertise, they inevitably show some nice old pine forest and talk about how much they care about the environment. Cigarettes cause death, so their ads show life, like all the ads with happy smiling healthy people exercising outdoors. And so on.

When the Macintosh first came out, there was no software available for it. So obviously, Apple created a giant glossy catalog listing all the great software that was "available". Half of the items listed said, in fine print, "under development," and the other half couldn't be had for love or money. Some were such lame products nobody would buy them. But even having a thick glossy catalog with one software "product" per page described in glowing prose couldn't disguise the fact that you just could not buy a word processor or spreadsheet to run on your 128KB Macintosh. There were similar "software product guides" for NeXT and BeOS. (Attention, NeXT and BeOS bigots: I don't need any flak about your poxy operating systems, OK? Write your own column.) The only thing a software product guide tells you is that there is no software available for the system. When you see one of these beasts, run fleeing in the opposite direction.

Amiga, Atari ST, Gem, IBM TopView, NeXT, BeOS, Windows CE, General Magic, the list of failed "new platforms" goes on and on. Because they are platforms, they are, by definition, not very interesting in and of themselves without juicy software to run on them. But, with very few exceptions (and I'm sure I'll get a whole host of email from tedious supporters of arcane and unloved platforms like the Amiga or RSTS-11), no software developer with the least bit of common sense would intentionally write software for a platform with 100,000 users on a good day, like BeOS, when they could do the same amount of work and create software for a platform with 100,000,000 users, like Windows. The fact that anybody writes software for those oddball systems at all proves that the profit motive isn't everything: religious fervor is still alive and well. Good for you, darling. You wrote a nice microEmacs clone for the Timex Sinclair 1000. Bravo. Here's a quarter, buy yourself a treat.

So. If you're in the platform creation business, you are probably going to suffer from what is commonly known as the chicken and egg problem. Nobody is going to buy your platform until there's good software that runs on it, and nobody is going to write software until you have a big installed base. Ooops. It's sort of like a Gordian Knot, although a Gordian Death Spiral might be more descriptive.

The chicken and egg problem, and variants thereof, is the most important element of strategy to understand. Well, OK, you can probably live without understanding it: Steve Jobs practically made a career out of not understanding the chicken and egg problem, twice. But the rest of us don't have Jobs' Personal Reality Distortion Field at our disposal, so we'll have to buckle down and study hard.

Lesson one. The classic domain of chicken and egg problems is in software platforms. But here's another chicken and egg problem: every month, millions of credit card companies mail out zillions of bills to consumers in the mail. People write paper checks, stuff them in trillions of envelopes, and mail them back. The envelopes are put in big boxes and taken to countries where labor is cheap to be opened and processed. But the whole operation costs quite a bit: the last figure I heard was that it is more than $1 per bill.

To us Internet wise-guys, that's a joke. "Email me my bill", you say. "I'll pay it online!" You say. "It'll only cost, say, 1/100000th of a penny. You'll save millions" Or something like that.

And you're right. So a lot of companies have tried to get into this field, which is technically known as Bill Presentment. One example is (guess who) Microsoft. Their solution, TransPoint, looks like this: it's a web site. You go there, and it shows you your bills. You pay them.

So, now, if you get your bills on this Microsoft system, you have to visit the web page every few days to see if any bills have arrived so you don't miss them. If you get, say, 10 bills a month, this might not be too big a hassle. Therein lies the other problem: there are only a small handful of merchants that will bill you over this system. So for all your other bills, you'll have to go elsewhere.

End result? It's not worth it. I would be surprised if 10,000 people are using this system. Now, Microsoft has to go to merchants and say, "bill your customers over our system!" And the merchants will say, "OK! How much will it cost?" And Microsoft will say, "50 cents! But it's a lot cheaper than $1!" And the merchants will say, "OK. Anything else?" And Microsoft will say, "Oh yes, it will cost you about $250,000 to set up the software, connect our systems to your systems, and get everything working."

And since Microsoft has so few dang users on this system, it's hard to imagine why anyone would pay $250,000 to save 50 cents on 37 users. Aha! The chicken and egg problem has reared its ugly head! Customers won't show up until you have merchants, and merchants won't show up until you have customers! Eventually, Microsoft is just going to spend their way out of this predicament. For smaller companies, that's not an option. So what can you do?

Software platforms actually gives us some nice hints as to how to roast your chicken and egg problem. Let's look a bit at the history of personal computer software platforms in the years since the IBM-PC came out; maybe we'll discover something!

Most people think that the IBM-PC required PC-DOS. Not true. When the IBM-PC first came out, you had a choice of three operating systems: PC-DOS, XENIX (a wimpy 8 bit version of UNIX published by, and I am not making this up, Microsoft), and something called UCSD P-System, which was, if you can believe this, just like Java: nice, slow, portable bytecodes, about 20 years before Java.

Now, most people have never heard of XENIX or UCSD's weirdo stuff. You kids today probably think that this is because Microsoft took over the market for dinky operating systems through marketing muscle or something. Absolutely not true; Microsoft was tiny in those days. The company with the marketing muscle was Digital Research, which had a different operating system. So, why was PC-DOS the winner of the three way race?

Before the PC, the only real operating system you could get was CP/M, although the market for CP/M based computers, which cost about $10,000, was too small. They were cranky and expensive and not very user friendly. But those who did buy them, did so to use as word processors, because you could get a pretty good word processor called WordStar for CP/M, and the Apple II just could not do word processing (it didn't have lower case, to begin with).

Now, here's a little known fact: even DOS 1.0 was designed with a CP/M backwards compatibility mode built in. Not only did it have its own spiffy new programming interface, known to hard core programmers as INT 21, but it fully supported the old CP/M programming interface. It could almost run CP/M software. In fact, WordStar was ported to DOS by changing one single byte in the code. (Real Programmers can tell you what that byte was, I've long since forgotten).

That bears mentioning again. WordStar was ported to DOS by changing one single byte in the code. Let that sink in.

There.

Got it?

DOS was popular because it had software from day one. And it had software because Tim Paterson had thought to include a CP/M compatibility feature in it, because way back in the dark ages somebody was smart about chicken and egg problems.

Fast forward. In the entire history of the PC platform, there have only been two major paradigm shifts that took along almost every PC user: we all switched to Windows 3.x, and then we all switched to Windows 95. Only a tiny number of people ever switched to anything else on the way. Microsoft conspiracy to take over the world? Fine, you're welcome to think that. I think it's for another, more interesting reason, which just comes back to the chicken and the egg.

We all switched to Windows 3.x. The important clue in that sentence is the 3. Why didn't we all switch to Windows 1.0? Or Windows 2.0? Or Windows 286 or Windows 386 which followed? Is it because it takes Microsoft five releases to "get it right"? No.

The actual reason was even more subtle than that, and it has to do with a very arcane hardware features that first showed up on the Intel 80386 chip which Windows 3.0 required.

  • Feature one: old DOS programs put things on the screen by writing directly to memory locations which corresponded to character cells on the screen. This was the only way to do output fast enough to make your program look good. But Windows ran in graphics mode. On older Intel chips, the Microsoft engineers had no choice but to flip into full screen mode when they were running DOS programs. But on the 80386, they could set up virtual memory blocks and set interrupts so that the operating system was notified whenever a program tried to write to screen memory. Windows could then write the equivalent text into a graphical window on the screen instantly.
  • Feature two: old DOS programs assumed they had the run of the chip. As a result, they didn't play well together. But the Intel 80386 had the ability to create "virtual" PCs, each of them acting like a complete 8086, so old PC programs could pretend like they had the computer to themselves, even while other programs were running and, themselves, pretending they had the whole computer to themselves.

So Windows 3.x on Intel 80386s was the first version that could run multiple DOS programs respectably. (Technically, Windows 386 could too, but 80386s were rare and expensive until about the time that Windows 3.0 came out.) Windows 3.0 was the first version that could actually do a reasonable job running all your old software.

Windows 95? No problem. Nice new 32 bit API, but it still ran old 16 bit software perfectly. Microsoft obsessed about this, spending a big chunk of change testing every old program they could find with Windows 95. Jon Ross, who wrote the original version of SimCity for Windows 3.x, told me that he accidentally left a bug in SimCity where he read memory that he had just freed. Yep. It worked fine on Windows 3.x, because the memory never went anywhere. Here's the amazing part: On beta versions of Windows 95, SimCity wasn't working in testing. Microsoft tracked down the bug and added specific code to Windows 95 that looks for SimCity. If it finds SimCity running, it runs the memory allocator in a special mode that doesn't free memory right away. That's the kind of obsession with backward compatibility that made people willing to upgrade to Windows 95.

You should be starting to get some ideas about how to break the chicken and egg problem: provide a backwards compatibility mode which either delivers a truckload of chickens, or a truckload of eggs, depending on how you look at it, and sit back and rake in the bucks.

Ah. Now back to bill presentment. Remember bill presentment? The chicken-egg problem is that you can only get your Con Ed bills, so you won't use the service. How can you solve it? Microsoft couldn't figure it out. PayMyBills.com (and a half dozen other Silicon Valley startups) all figured it out at the same time. You provide a backwards compatibility mode: if the merchant won't support the system, just get the merchant to mail their damn paper bills to University Avenue, in Palo Alto, where a bunch of actual human beings will open them and scan them in. Now you can get all your bills on their web site. Since every merchant on earth is available on the system, customers are happy to use it, even if it is running in this weird backwards compatibility mode where stupid Visa member banks send the bill electronically to a printer, print it out on paper, stuff it in an envelope, ship it 1500 miles to California, where it is cut open, the stupid flyers harping worthless "free" AM clock radios that actually cost $9.95 are thrown into a landfill somewhere, and the paper bill is scanned back into a computer and stuck up on the web where it should have been sent in the first place. But the stupid backwards compatibility mode will eventually go away, because PayMyBills.com, unlike Microsoft, can actually get customers to use their system, so pretty soon they'll be able to go to the stupid Visa member banks and say, "hey, I've got 93,400 of your customers. Why don't you save yourselves $93,400 each month with a direct wire connection to me?" And suddenly PayMyBills.com is very profitable while Microsoft is still struggling to sign up their second electric utility, maybe one serving Georgia would be a nice change of pace.

Companies that fail to recognize the Chicken and Egg problem can be thought of as boil the ocean companies: their business plan requires 93,000,000 humans to cooperate with their crazy business scheme before it actually works. One of the most outrageously stupid ideas I ever encountered was called ActiveNames. Their boneheaded idea was that everybody in the world would install a little add-in to their email client which looked up people's names on their central servers to get the actual email address. Then instead of telling people that your email address is kermit@sesame-street.com, you would tell them that your ActiveName is "spolsky", and if they want to email you, they need to install this special software. Bzzzzzt. Wrong answer. I can't even begin to list all the reasons this idea is never going to work.

Conclusion: if you're in a market with a chicken and egg problem, you better have a backwards-compatibility answer that dissolves the problem, or it's going to take you a loooong time to get going (like, forever).

There are a lot of other companies that recognized the chicken and egg problem face on and defeated it intelligently. When Transmeta unveiled their new CPU, it was the first time in a long time that a company that was not Intel finally admitted that if you're a CPU, and you want a zillion people to buy you, you gotta run x86 code. This after Hitachi, Motorola, IBM, MIPS, National Semiconductor, and who knows how many other companies deceived themselves into thinking that they had the right to invent a new instruction set. The Transmeta architecture assumes from day one that any business plan that calls for making a computer that doesn't run Excel is just not going anywhere.


Strategy Letter III: Let Me Go Back!

Saturday, June 03, 2000

When you're trying to get people to switch from a competitor to your product, you need to understand barriers to entry, and you need to understand them a lot better than you think, or people won't switch and you'll be waiting tables.

In an earlier letter, I wrote about the difference between two kinds of companies: the Ben and Jerry's kind of company which is trying to take over from established competition, versus the Amazon.com kind of company which is trying a "land grab" in a new field where there is no established competition. When I worked on Microsoft Excel in the early 90's, it was a card-carrying member of the Ben and Jerry's camp. Lotus 123, the established competitor, had an almost complete monopoly in the market for spreadsheets. Sure, there were new users buying computers who started out with Excel, but for the most part, if Microsoft wanted to sell spreadsheets, they were going to have to get people to switch.

The most important thing to do when you're in this position is to admit it. Some companies can't even do this. The management at my last employer, Juno, was unwilling to admit that AOL had already achieved a dominant position. They spoke of the "millions of people not yet online." They said that "in every market, there is room for two players: Time and Newsweek, Coke and Pepsi, etc." The only thing they wouldn't say is "we have to get people to switch away from AOL." I'm not sure what they were afraid of. Perhaps they thought they were afraid to "wake up the sleeping bear". When one of Juno's star programmers (no, not me) had the chutzpah, the unmitigated gall to ask a simple question at a company meeting: "Why aren't we doing more to get AOL users to switch?" they hauled him off, screamed at him for an hour, and denied him a promotion he had been promised. (Guess who took his talent elsewhere?)

There's nothing wrong with being in a market that has established competition. In fact, even if your product is radically new, like eBay, you probably have competition: garage sales! Don't stress too much. If your product is better in some way, you actually have a pretty good chance of getting people to switch. But you have to think strategically about it, and thinking strategically means thinking one step beyond the obvious.

The only strategy in getting people to switch to your product is to eliminate barriers. Imagine that it's 1991. The dominant spreadsheet, with 100% market share, is Lotus 123. You're the product manager for Microsoft Excel. Ask yourself: what are the barriers to switching? What keeps users from becoming Excel customers tomorrow?

Barrier
1. They have to know about Excel and know that it's better
2. They have to buy Excel
3. They have to buy Windows to run Excel
4. They have to convert their existing spreadsheets from 123 to Excel
5. They have to rewrite their keyboard macros which won't run in Excel
6. They have to learn a new user interface
7. They need a faster computer with more memory

And so on, and so on. Think of these barriers as an obstacle course that people have to run before you can count them as your customers. If you start out with a field of 1000 runners, about half of them will trip on the tires; half of the survivors won't be strong enough to jump the wall; half of those survivors will fall off the rope ladder into the mud, and so on, until only 1 or 2 people actually overcome all the hurdles. With 8 or 9 barriers, everybody will have one non-negotiable deal killer.

This calculus means that eliminating barriers to switching is the most important thing you have to do if you want to take over an existing market, because eliminating just one barrier will likely double your sales. Eliminate two barriers, and you'll double your sales again. Microsoft looked at the list of barriers and worked on all of them:

Barrier | Solution
1. They have to know about Excel and know that it's better | Advertise Excel, send out demo disks, and tour the country showing it off
2. They have to buy Excel | Offer a special discount for former 123 users to switch to Excel
3. They have to buy Windows to run Excel | Make a runtime version of Windows which ships free with Excel
4. They have to convert their existing spreadsheets from 123 to Excel | Give Excel the capability to read 123 spreadsheets
5. They have to rewrite their keyboard macros which won't run in Excel | Give Excel the capability to run 123 macros
6. They have to learn a new user interface | Give Excel the ability to understand Lotus keystrokes, in case you were used to the old way of doing things
7. They need a faster computer with more memory | Wait for Moore's law to solve the problem of computer power

And it worked pretty well. By incessant pounding on eliminating barriers, they slowly pried some market share away from Lotus.

One thing you see a lot when there is a transition from an old monopoly to a new monopoly is that there is a magic "tipping point": one morning, you wake up and your product has 80% market share instead of 20% market share. This flip tends to happen very quickly (VisiCalc to 123 to Excel, WordStar to WordPerfect to Word, Mosaic to Netscape to Internet Explorer, dBase to Access, and so on). It usually happens because the very last barrier to entry has fallen and suddenly it's logical for everyone to switch.

Obviously, it's important to work on fixing the obvious barriers to entry, but once you think you've addressed those, you need to figure out what the non-so-obvious ones are. And this is where strategy becomes tricky, because there are some non-obvious things that keep people from switching.

Here's an example. This summer I'm spending most of my time in a house near the beach, but my bills still go to the apartment in New York City. And I travel a lot. There's a nice web service, PayMyBills.com, which is supposed to simplify your life: you have all your bills sent to them, and they scan them and put them on the web for you to see wherever you may be.

Now, PayMyBills costs about $9 a month, which sounds reasonable, and I would consider using it, but in the past, I've had pretty bad luck with financial services on the Internet, like Datek, which made so many arithmetic mistakes in my statements I couldn't believe they were licensed. So I'm willing to try PayMyBills, but if I don't like it, I want to be able to go back to the old way.

The trouble is, after I use PayMyBills, if I don't like it, I need to call every damn credit card company and change my address again. That's a lot of work. And so the fear of how hard it will be to switch back is keeping me from using their service. Earlier I called this "stealth lock-in," and sort of praised it, but if potential customers figure it out, oh boy are you in trouble.

That's the barrier to entry. Not how hard it is to switch in: it's how hard it might be to switch out.

And this reminded me of Excel's tipping point, which happened around the time of Excel 4.0. And the biggest reason was that Excel 4.0 was the first version of Excel that could write Lotus spreadsheets transparently.

Yep, you heard me. Write. Not read. It turns out that what was stopping people from switching to Excel was that everybody else they worked with was still using Lotus 123. They didn't want a product that would create spreadsheets that nobody else could read: a classic Chicken and Egg problem. When you're the lone Excel fan in a company where everyone else is using 123, even if you love Excel, you can't switch until you can participate in the 123 ecology.

To take over a market, you have to address every barrier to entry. If you forget just one barrier which trips up 50% of your potential customers, then by definition, you can't have more than 50% market share, and you will never displace the dominant player, and you'll be stuck on the sad (omelet) side of chicken and egg problems.

The trouble is that most managers only think about strategy one step at a time, like chess players who refuse to think one move ahead. Most of them will say, "it's important to let people convert into your product, but why should I waste my limited engineering budget letting people convert out?"

That's a childish approach to strategy. It reminds me of independent booksellers, who said "why should I make it comfortable for people to read books in my store? I want them to buy the books!" And then one day Barnes and Nobles puts couches and cafes in the stores and practically begged people to read books in their store without buying them. Now you've got all these customers sitting in their stores for hours at a time, mittengrabben all the books with their filthy hands, and the probability that they find something they want to buy is linearly proportional to the amount of time they spend in the store, and even the dinkiest Barnes and Nobles superstore in Iowa City rakes in hundreds of dollars a minute while the independent booksellers are going out of business. Honey, Shakespeare and Company on Manhattan's Upper West Side did not close because Barnes and Nobles had cheaper prices, it closed because Barnes and Nobles had more human beings in the building.

The mature approach to strategy is not to try to force things on potential customers. If somebody isn't even your customer yet, trying to lock them in just isn't a good idea. When you have 100% market share, come talk to me about lock-in. Until then, if you try to lock them in now, it's too early, and if any customer catches you in the act, you'll just wind up locking them out. Nobody wants to switch to a product that is going to eliminate their freedom in the future.

Let's take a more current example: ISPs, a highly competitive market. Something that virtually no ISP offers is the ability to get your email forwarded to another email address after you quit their service. This is small-minded thinking of the worst sort, and I'm pretty surprised nobody has figured it out. If you're a small ISP trying to get people to switch, they are going to be worrying about the biggest barrier: telling all their friends their new email address. So they won't even want to try your service. If they do try it, they won't tell their friends the new address for a while, just in case it doesn't work out. Which means they won't be getting much email at the new address, which means they won't really be trying out the service and seeing how much better they like it. Lose-lose.

Now suppose one brave ISP would make the following promise: "Try us. If you don't like us, we'll keep your email address functioning, and we'll forward your email for free to any other ISP. For life. Hop around from ISP to ISP as many times as you want, just let us know, and we'll be your permanent forwarding service."

Of course, the business managers would have fits. Why should we make it easy for customers to leave the service? That's because they are short sighted. These are not your customers now. Try to lock them in before they become your customers, and you'll just lock them out. But if you make an honest promise that it will be easy to back out of the service if they're not happy, and suddenly you eliminate one more barrier to entry. And, as we learned, eliminating even a single barrier to entry can have a dramatic effect on conversions, and over time, when you knock down that last barrier to entry, people will start flooding in, and life will be good for a while. Until somebody does the same thing to you.


Strategy Letter IV: Bloatware and the 80/20 Myth

Friday, March 23, 2001

Version 5.0 of Microsoft's flagship spreadsheet program Excel came out in 1993. It was positively huge: it required a whole 15 megabytes of hard drive space. In those days we could still remember our first 20MB PC hard drives (around 1985) and so 15MB sure seemed like a lot.

By the time Excel 2000 came out, it required a whopping 146MB ... almost a tenfold increase! Dang those sloppy Microsoft programmers, right?

Wrong.

I'll bet you think I'm going to write one of those boring articles you see all over the net bemoaning "bloatware". Whine whine whine, this stuff is so bloated, oh woe is me, edlin and vi are so much better than Word and Emacs because they are svelte, etc.

Ha ha! I tricked you! I'm not going to write that article again, because it's not true.

In 1993, given the cost of hard drives in those days, Microsoft Excel 5.0 took up about $36 worth of hard drive space.

In 2000, given the cost of hard drives in 2000, Microsoft Excel 2000 takes up about $1.03 in hard drive space.

(These figures are adjusted for inflation and based on hard drive price data from here.)

In real terms, it's almost like Excel is actually getting smaller!

What is bloatware, exactly? The Jargon File snidely defines it as "software that provides minimal functionality while requiring a disproportionate amount of diskspace and memory. Especially used for application and OS upgrades. This term is very common in the Windows/NT world. So is its cause."

I guess those guys just hate Windows. I haven't run out of memory in more than a decade, ever since virtual memory appeared in Windows 386 (1989). And hard drive space is down to $0.0071 per megabyte and still plummeting like a sheep learning to fly by jumping out of a tree.

Maybe Linus ?hellip;kerlund can explain it. On his web page, he writes, "The big disadvantage of using these bloated programs is that you have to load this very large program, even if you just want to accomplish one tiny little thing. It eats up all your memory... you're not using your system in an efficient way. You make the system seem more inefficient than it really is, and this is totally unnecessary."

Ohhh. It eats up all your memory. I see. Actually, well, no, it doesn't. Ever since Windows 1.0, in 1987, the operating system only loads pages as they are used. If you have a 15MB executable and you only use code that spans 2MB worth of pages, you will only ever load 2MB from disk to RAM. In fact if you have a modern version of Windows, the OS will automatically rearrange those pages on the hard drive so that they're consecutive, which makes the program start even faster next time.

And I don't think anyone will deny that on today's overpowered, under-priced computers, loading a huge program is still faster than loading a small program was even 5 years ago. So what's the problem?

RA Downes gives us a clue. It looks like he spent hours dissecting a small Microsoft utility, apparently enraged that it was a whole megabyte in size. (That's 3.15 cents of hard drive space at the time he wrote the article). In his opinion, the program should have been around 95% smaller. The joke is that the utility he dissected is something called RegClean, which you've probably never heard of. This is a program that goes through your Windows registry looking for things that aren't being used and deleting them. You have to be a little bit on the obsessive-compulsive side to care about cleaning up unused parts of your registry. So I'm starting to suspect that fretting about bloatware is more of a mental health problem than a software problem.

In fact there are lots of great reasons for bloatware. For one, if programmers don't have to worry about how large their code is, they can ship it sooner. And that means you get more features, and features make your life better (when you use them) and don't usually hurt (when you don't). If your software vendor stops, before shipping, and spends two months squeezing the code down to make it 50% smaller, the net benefit to you is going to be imperceptible. Maybe, just maybe, if you tend to keep your hard drive full, that's one more Duran Duran MP3 you can download. But the loss to you of waiting an extra two months for the new version is perceptible, and the loss to the software company that has to give up two months of sales is even worse.

A lot of software developers are seduced by the old "80/20" rule. It seems to make a lot of sense: 80% of the people use 20% of the features. So you convince yourself that you only need to implement 20% of the features, and you can still sell 80% as many copies.

Unfortunately, it's never the same 20%. Everybody uses a different set of features. In the last 10 years I have probably heard of dozens of companies who, determined not to learn from each other, tried to release "lite" word processors that only implement 20% of the features. This story is as old as the PC. Most of the time, what happens is that they give their program to a journalist to review, and the journalist reviews it by writing their review using the new word processor, and then the journalist tries to find the "word count" feature which they need because most journalists have precise word count requirements, and it's not there, because it's in the "80% that nobody uses," and the journalist ends up writing a story that attempts to claim simultaneously that lite programs are good, bloat is bad, and I can't use this damn thing 'cause it won't count my words. If I had a dollar for every time this has happened I would be very happy.

When you start marketing your "lite" product, and you tell people, "hey, it's lite, only 1MB," they tend to be very happy, then they ask you if it has their crucial feature, and it doesn't, so they don't buy your product.

Bottom line: if your strategy is "80/20", you're going to have trouble selling software. That's just reality. This strategy is as old as the software industry itself and it just doesn't pay; what's surprising is how many executives at fast companies think that it's going to work.

Jamie Zawinski says it best, discussing the original version of Netscape that changed the world. "Convenient though it would be if it were true, Mozilla [Netscape 1.0] is not big because it's full of useless crap. Mozilla is big because your needs are big. Your needs are big because the Internet is big. There are lots of small, lean web browsers out there that, incidentally, do almost nothing useful. But being a shining jewel of perfection was not a goal when we wrote Mozilla."


Strategy Letter V

Wednesday, June 12, 2002

When I was in college I took two intro economics courses: macroeconomics and microeconomics. Macro was full of theories like "low unemployment causes inflation" that never quite stood up to reality. But the micro stuff was both cool and useful. It was full of interesting concepts about the relationships between supply and demand that really did work. For example, if you have a competitor who lowers their prices, the demand for your product will go down unless you match them.

In today's episode, I'll show how one of those concepts explains a lot about some familiar computer companies. Along the way, I noticed something interesting about open source software, which is this: most of the companies spending big money to develop open source software are doing it because it's a good business strategy for them, not because they suddenly stopped believing in capitalism and fell in love with freedom-as-in-speech.

Every product in the marketplace has substitutes and complements. A substitute is another product you might buy if the first product is too expensive. Chicken is a substitute for beef. If you're a chicken farmer and the price of beef goes up, the people will want more chicken, and you will sell more.

A complement is a product that you usually buy together with another product. Gas and cars are complements. Computer hardware is a classic complement of computer operating systems. And babysitters are a complement of dinner at fine restaurants. In a small town, when the local five star restaurant has a two-for-one Valentine's day special, the local babysitters double their rates. (Actually, the nine-year-olds get roped into early service.)

All else being equal, demand for a product increases when the prices of its complements decrease.

Let me repeat that because you might have dozed off, and it's important. Demand for a product increases when the prices of its complements decrease. For example, if flights to Miami become cheaper, demand for hotel rooms in Miami goes up -- because more people are flying to Miami and need a room. When computers become cheaper, more people buy them, and they all need operating systems, so demand for operating systems goes up, which means the price of operating systems can go up.

At this point, it's pretty common for people to try to confuse things by saying, "aha! But Linux is FREE!" OK. First of all, when an economist considers price, they consider the total price, including some intangible things like the time it takes to set up, reeducate everyone, and convert existing processes. All the things that we like to call "total cost of ownership."

Secondly, by using the free-as-in-beer argument, these advocates try to believe that they are not subject to the rules of economics because they've got a nice zero they can multiply everything by. Here's an example. When Slashdot asked Linux developer Moshe Bar if future Linux kernels would be compatible with existing device drivers, he said that they didn't need to. "Proprietary software goes at the tariff of US$ 50-200 per line of debugged code. No such price applies to OpenSource software." Moshe goes on to claim that it's OK for every Linux kernel revision to make all existing drivers obsolete, because the cost of rewriting all those existing drivers is zero. This is completely wrong. He's basically claiming that spending a small amount of programming time making the kernel backwards compatible is equivalent to spending a huge amount of programming time rewriting every driver, because both numbers are multiplied by their "cost," which he believes to be zero. This is a prima facie fallacy. The thousands or millions of developer hours it takes to revise every existing device driver are going to have to come at the expense of something. And until that's done, Linux will be once again handicapped in the marketplace because it doesn't support existing hardware. Wouldn't it be better to use all that "zero cost" effort making Gnome better? Or supporting new hardware?

Debugged code is NOT free, whether proprietary or open source. Even if you don't pay cash dollars for it, it has opportunity cost, and it has time cost. There is a finite amount of volunteer programming talent available for open source work, and each open source project competes with each other open source project for the same limited programming resource, and only the sexiest projects really have more volunteer developers than they can use. To summarize, I'm not very impressed by people who try to prove wild economic things about free-as-in-beer software, because they're just getting divide-by-zero errors as far as I'm concerned.

Open source is not exempt from the laws of gravity or economics. We saw this with Eazel, ArsDigita, The Company Formerly Known as VA Linux and a lot of other attempts. But something is still going on which very few people in the open source world really understand: a lot of very large public companies, with responsibilities to maximize shareholder value, are investing a lot of money in supporting open source software, usually by paying large teams of programmers to work on it. And that's what the principle of complements explains.

Once again: demand for a product increases when the price of its complements decreases. In general, a company's strategic interest is going to be to get the price of their complements as low as possible. The lowest theoretically sustainable price would be the "commodity price" -- the price that arises when you have a bunch of competitors offering indistinguishable goods. So:


The Absolute Minimum Every Software Developer Absolutely, Positively Must Know About Unicode and Character Sets (No Excuses!)

Wednesday, October 08, 2003

Ever wonder about that mysterious Content-Type tag? You know, the one you're supposed to put in HTML and you never quite know what it should be?

Did you ever get an email from your friends in Bulgaria with the subject line "???? ?????? ??? ????"?

I've been dismayed to discover just how many software developers aren't really completely up to speed on the mysterious world of character sets, encodings, Unicode, all that stuff. A couple of years ago, a beta tester for

But it won't. When I discovered that the popular web development tool PHP has almost complete ignorance of character encoding issues, blithely using 8 bits for characters, making it darn near impossible to develop good international web applications, I thought, enough is enough.

So I have an announcement to make: if you are a programmer working in 2003 and you don't know the basics of characters, character sets, encodings, and Unicode, and I catch you, I'm going to punish you by making you peel onions for 6 months in a submarine. I swear I will.

And one more thing:

IT'S NOT THAT HARD.

In this article I'll fill you in on exactly what every working programmer should know. All that stuff about "plain text = ascii = characters are 8 bits" is not only wrong, it's hopelessly wrong, and if you're still programming that way, you're not much better than a medical doctor who doesn't believe in germs. Please do not write another line of code until you finish reading this article.

Before I get started, I should warn you that if you are one of those rare people who knows about internationalization, you are going to find my entire discussion a little bit oversimplified. I'm really just trying to set a minimum bar here so that everyone can understand what's going on and can write code that has a hope of working with text in any language other than the subset of English that doesn't include words with accents. And I should warn you that character handling is only a tiny portion of what it takes to create software that works internationally, but I can only write about one thing at a time so today it's character sets.

A Historical Perspective

The easiest way to understand this stuff is to go chronologically.

You probably think I'm going to talk about very old character sets like EBCDIC here. Well, I won't. EBCDIC is not relevant to your life. We don't have to go that far back in time.

Back in the semi-olden days, when Unix was being invented and K&R were writing

And all was good, assuming you were an English speaker.

Because bytes have room for up to eight bits, lots of people got to thinking, "gosh, we can use the codes 128-255 for our own purposes." The trouble was,

Eventually this OEM free-for-all got codified in the ANSI standard. In the ANSI standard, everybody agreed on what to do below 128, which was pretty much the same as ASCII, but there were lots of different ways to handle the characters from 128 and on up, depending on where you lived. These different systems were called code pages. So for example in Israel DOS used a code page called 862, while Greek users used 737. They were the same below 128 but different from 128 up, where all the funny letters resided. The national versions of MS-DOS had dozens of these code pages, handling everything from English to Icelandic and they even had a few "multilingual" code pages that could do Esperanto and Galician on the same computer! Wow! But getting, say, Hebrew and Greek on the same computer was a complete impossibility unless you wrote your own custom program that displayed everything using bitmapped graphics, because Hebrew and Greek required different code pages with different interpretations of the high numbers.

Meanwhile, in Asia, even more crazy things were going on to take into account the fact that Asian alphabets have thousands of letters, which were never going to fit into 8 bits. This was usually solved by the messy system called DBCS, the "double byte character set" in which some letters were stored in one byte and others took two. It was easy to move forward in a string, but dang near impossible to move backwards. Programmers were encouraged not to use s++ and s-- to move backwards and forwards, but instead to call functions such as Windows' AnsiNext and AnsiPrev which knew how to deal with the whole mess.

But still, most people just pretended that a byte was a character and a character was 8 bits and as long as you never moved a string from one computer to another, or spoke more than one language, it would sort of always work. But of course, as soon as the Internet happened, it became quite commonplace to move strings from one computer to another, and the whole mess came tumbling down. Luckily, Unicode had been invented.

Unicode

Unicode was a brave effort to create a single character set that included every reasonable writing system on the planet and some make-believe ones like Klingon, too. Some people are under the misconception that Unicode is simply a 16-bit code where each character takes 16 bits and therefore there are 65,536 possible characters. This is not, actually, correct. It is the single most common myth about Unicode, so if you thought that, don't feel bad.

In fact, Unicode has a different way of thinking about characters, and you have to understand the Unicode way of thinking of things or nothing will make sense.

Until now, we've assumed that a letter maps to some bits which you can store on disk or in memory:

A -> 0100 0001

In Unicode, a letter maps to something called a code point which is still just a theoretical concept. How that code point is represented in memory or on disk is a whole nuther story.

In Unicode, the letter A is a platonic ideal. It's just floating in heaven:

A

This platonic A is different than B, and different from a, but the same as A and A and A. The idea that A in a Times New Roman font is the same character as the A in a Helvetica font, but different from "a" in lower case, does not seem very controversial, but in some languages just figuring out what a letter is can cause controversy. Is the German letter ?Yuml; a real letter or just a fancy way of writing ss? If a letter's shape changes at the end of the word, is that a different letter? Hebrew says yes, Arabic says no. Anyway, the smart people at the Unicode consortium have been figuring this out for the last decade or so, accompanied by a great deal of highly political debate, and you don't have to worry about it. They've figured it all out already.

Every platonic letter in every alphabet is assigned a magic number by the Unicode consortium which is written like this: U+0645. This magic number is called a code point. The U+ means "Unicode" and the numbers are hexadecimal. U+FEC9 is the Arabic letter Ain. The English letter A would be U+0041. You can find them all using the charmap utility on Windows 2000/XP or visiting the Unicode web site.

There is no real limit on the number of letters that Unicode can define and in fact they have gone beyond 65,536 so not every unicode letter can really be squeezed into two bytes, but that was a myth anyway.

OK, so say we have a string:

Hello

which, in Unicode, corresponds to these five code points:

U+0048 U+0065 U+006C U+006C U+006F.

Just a bunch of code points. Numbers, really. We haven't yet said anything about how to store this in memory or represent it in an email message.

Encodings

That's where encodings come in.

The earliest idea for Unicode encoding, which led to the myth about the two bytes, was, hey, let's just store those numbers in two bytes each. So Hello becomes

00 48 00 65 00 6C 00 6C 00 6F

Right? Not so fast! Couldn't it also be:

48 00 65 00 6C 00 6C 00 6F 00 ?

Well, technically, yes, I do believe it could, and, in fact, early implementors wanted to be able to store their Unicode code points in high-endian or low-endian mode, whichever their particular CPU was fastest at, and lo, it was evening and it was morning and there were already two ways to store Unicode. So the people were forced to come up with the bizarre convention of storing a FE FF at the beginning of every Unicode string; this is called a Unicode Byte Order Mark and if you are swapping your high and low bytes it will look like a FF FE and the person reading your string will know that they have to swap every other byte. Phew. Not every Unicode string in the wild has a byte order mark at the beginning.

For a while it seemed like that might be good enough, but programmers were complaining. "Look at all those zeros!" they said, since they were Americans and they were looking at English text which rarely used code points above U+00FF. Also they were liberal hippies in California who wanted to conserve (sneer). If they were Texans they wouldn't have minded guzzling twice the number of bytes. But those Californian wimps couldn't bear the idea of doubling the amount of storage it took for strings, and anyway, there were already all these doggone documents out there using various ANSI and DBCS character sets and who's going to convert them all? Moi? For this reason alone most people decided to ignore Unicode for several years and in the meantime things got worse.

Thus was invented the brilliant concept of UTF-8. UTF-8 was another system for storing your string of Unicode code points, those magic U+ numbers, in memory using 8 bit bytes. In UTF-8, every code point from 0-127 is stored in a single byte. Only code points 128 and above are stored using 2, 3, in fact, up to 6 bytes.

This has the neat side effect that English text looks exactly the same in UTF-8 as it did in ASCII, so Americans don't even notice anything wrong. Only the rest of the world has to jump through hoops. Specifically, Hello, which was U+0048 U+0065 U+006C U+006C U+006F, will be stored as 48 65 6C 6C 6F, which, behold! is the same as it was stored in ASCII, and ANSI, and every OEM character set on the planet. Now, if you are so bold as to use accented letters or Greek letters or Klingon letters, you'll have to use several bytes to store a single code point, but the Americans will never notice. (UTF-8 also has the nice property that ignorant old string-processing code that wants to use a single 0 byte as the null-terminator will not truncate strings).

So far I've told you three ways of encoding Unicode. The traditional store-it-in-two-byte methods are called UCS-2 (because it has two bytes) or UTF-16 (because it has 16 bits), and you still have to figure out if it's high-endian UCS-2 or low-endian UCS-2. And there's the popular new UTF-8 standard which has the nice property of also working respectably if you have the happy coincidence of English text and braindead programs that are completely unaware that there is anything other than ASCII.

There are actually a bunch of other ways of encoding Unicode. There's something called UTF-7, which is a lot like UTF-8 but guarantees that the high bit will always be zero, so that if you have to pass Unicode through some kind of draconian police-state email system that thinks 7 bits are quite enough, thank you it can still squeeze through unscathed. There's UCS-4, which stores each code point in 4 bytes, which has the nice property that every single code point can be stored in the same number of bytes, but, golly, even the Texans wouldn't be so bold as to waste that much memory.

And in fact now that you're thinking of things in terms of platonic ideal letters which are represented by Unicode code points, those unicode code points can be encoded in any old-school encoding scheme, too! For example, you could encode the Unicode string for Hello (U+0048 U+0065 U+006C U+006C U+006F) in ASCII, or the old OEM Greek Encoding, or the Hebrew ANSI Encoding, or any of several hundred encodings that have been invented so far, with one catch: some of the letters might not show up! If there's no equivalent for the Unicode code point you're trying to represent in the encoding you're trying to represent it in, you usually get a little question mark: ? or, if you're really good, a box. Which did you get? -> 锟?/p>

There are hundreds of traditional encodings which can only store some code points correctly and change all the other code points into question marks. Some popular encodings of English text are Windows-1252 (the Windows 9x standard for Western European languages) and ISO-8859-1, aka Latin-1 (also useful for any Western European language). But try to store Russian or Hebrew letters in these encodings and you get a bunch of question marks. UTF 7, 8, 16, and 32 all have the nice property of being able to store any code point correctly.

The Single Most Important Fact About Encodings

If you completely forget everything I just explained, please remember one extremely important fact. It does not make sense to have a string without knowing what encoding it uses. You can no longer stick your head in the sand and pretend that "plain" text is ASCII.

There Ain't No Such Thing As Plain Text.

If you have a string, in memory, in a file, or in an email message, you have to know what encoding it is in or you cannot interpret it or display it to users correctly.

Almost every stupid "my website looks like gibberish" or "she can't read my emails when I use accents" problem comes down to one naive programmer who didn't understand the simple fact that if you don't tell me whether a particular string is encoded using UTF-8 or ASCII or ISO 8859-1 (Latin 1) or Windows 1252 (Western European), you simply cannot display it correctly or even figure out where it ends. There are over a hundred encodings and above code point 127, all bets are off.

How do we preserve this information about what encoding a string uses? Well, there are standard ways to do this. For an email message, you are expected to have a string in the header of the form

Content-Type: text/plain; charset="UTF-8"

For a web page, the original idea was that the web server would return a similar Content-Type http header along with the web page itself -- not in the HTML itself, but as one of the response headers that are sent before the HTML page.

This causes problems. Suppose you have a big web server with lots of sites and hundreds of pages contributed by lots of people in lots of different languages and all using whatever encoding their copy of Microsoft FrontPage saw fit to generate. The web server itself wouldn't really know what encoding each file was written in, so it couldn't send the Content-Type header.

It would be convenient if you could put the Content-Type of the HTML file right in the HTML file itself, using some kind of special tag. Of course this drove purists crazy... how can you read the HTML file until you know what encoding it's in?! Luckily, almost every encoding in common use does the same thing with characters between 32 and 127, so you can always get this far on the HTML page without starting to use funny letters:

<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">

But that meta tag really has to be the very first thing in the <head> section because as soon as the web browser sees this tag it's going to stop parsing the page and start over after reinterpreting the whole page using the encoding you specified.

What do web browsers do if they don't find any Content-Type, either in the http headers or the meta tag? Internet Explorer actually does something quite interesting: it tries to guess, based on the frequency in which various bytes appear in typical text in typical encodings of various languages, what language and encoding was used. Because the various old 8 byte code pages tended to put their national letters in different ranges between 128 and 255, and because every human language has a different characteristic histogram of letter usage, this actually has a chance of working. It's truly weird, but it does seem to work often enough that na茂ve web-page writers who never knew they needed a Content-Type header look at their page in a web browser and it looks ok, until one day, they write something that doesn't exactly conform to the letter-frequency-distribution of their native language, and Internet Explorer decides it's Korean and displays it thusly, proving, I think, the point that Postel's Law about being "conservative in what you emit and liberal in what you accept" is quite frankly not a good engineering principle. Anyway, what does the poor reader of this website, which was written in Bulgarian but appears to be Korean (and not even cohesive Korean), do? He uses the View | Encoding menu and tries a bunch of different encodings (there are at least a dozen for Eastern European languages) until the picture comes in clearer. If he knew to do that, which most people don't.

For the latest version of CityDesk, the web site management software published by my company, we decided to do everything internally in UCS-2 (two byte) Unicode, which is what Visual Basic, COM, and Windows NT/2000/XP use as their native string type. In C++ code we just declare strings as wchar_t ("wide char") instead of char and use the wcs functions instead of the str functions (for example wcscat and wcslen instead of strcat and strlen). To create a literal UCS-2 string in C code you just put an L before it as so: L"Hello".

When CityDesk publishes the web page, it converts it to UTF-8 encoding, which has been well supported by web browsers for many years. That's the way all 29 language versions of Joel on Software are encoded and I have not yet heard a single person who has had any trouble viewing them.

This article is getting rather long, and I can't possibly cover everything there is to know about character encodings and Unicode, but I hope that if you've read this far, you know enough to go back to programming, using antibiotics instead of leeches and spells, a task to which I will leave you now.


The Book Club

Thursday, October 16, 2003

The following books have been featured in the Joel on Software Book Club.

Small Things Considered: Why There Is No Perfect Design
Henry Petroski
Featured October 2003

The imperfection of design is the theme of October's Book of the Month. Did you ever think about why calculators have 1, 2, and 3 on the bottom row while phones put those keys on the top row? Why did the high beam headlight switch migrate from a floorboard pedal to a toggle on the steering shaft? Whatever you're designing, from the error handling facilities of your software to the fat handle of a toothbrush which is highly ergonomic but can't fit in anyone's toothbrush holder, you have to trade off things that can't really be balanced against each other. And no matter what you do, you'll be subject to criticism, much of it valid.

Henry Petroski, who can write a 448 page book about the common pencil and make it fascinating, has done it again, this time with an excellent book about why there is no perfect design. All design is about tradeoffs, and if you don't believe me, this book offers dozens of examples from everyday life. It even offers a candidate for the best designed object on the planet (the three-legged plastic nubbin that keeps your pizza box lid from sticking to the cheese) and shows why even that is imperfect. Small Things Considered: Why There Is No Perfect Design is another great read and it's the first Joel on Software book of the month.

Buy it at Amazon


The //comment FAQ!

Thursday, December 09, 2004

Q. What is this book you keep going on about?

A. The people at Apress have asked me to serve as editor of a book tentatively titled //comment: The Best Software Essays of 2004.

Q. How are essays chosen?

A. Essays were nominated by the readers of Joel on Software. I also picked some of my own favorites.

The goal of this book is to encourage better writing about software by highlighting some of the best writing of the year. What I'm looking for is simple: good writing. Writing that is engaging, because it is funny, clear, or important, and writing that draws you in. I want the experience of reading about software to be better. I want software developers to learn how to write things that people want to read, because communicating is one of the most overlooked skills in software development.

The final decision of what to include is mine and I take full responsibility for the shoddy choices I've made.

If the first edition of this book sells well enough to justify a follow up next year, I'm hoping to have a committee.

To qualify, essays have to be written "around" 2004. Some are a little bit older, which will be ok for this first edition as long as they are still timely, but in future years I'll be a bit more strict.

Q. How is the book organized?

A. First I'll write a brief foreword.

Then the articles will appear in strict alphabetical order by author's last name.

Each article will be introduced by me, sometimes with one sentence, sometimes with a paragraph or two. This will be my only chance to interject my own opinion. After that, the author gets the last word. Yes, in many cases, I've chosen essays that I happen to disagree with vehemently. If they're good writing, I still want to publish them. My introduction may reflect my personal opinion but the author always gets the last word and I won't interfere with their writing.

I may insert editor's footnotes which are usually meant to explain technical concepts that may not be familiar to the average reader, and to explain things that were hyperlinks to outside websites or references to previous web articles in the original. Footnotes will not be used to add my editorial opinions. The original author can and should replace hyperlinks with footnotes explaining what's going on to the reader. My opinion will be confined to the introductory paragraphs. Editors footnotes will end with [--Ed.]

The first page of every essay will have an unnumbered footnote at the bottom which says "This work originally appeared on the web on [DATE] at http://..." and, if necessary, "... and has been [substantially] edited for this edition." In those rare cases where we're reprinting magazine articles instead of things on the web, we'll refer to the original magazine article in MLA style.

Finally there will be a big ol "about the authors" section at the back, also in alphabetical order, with up to a paragraph of biographical information about each author as submitted by the author themselves.

Q. Will essays be edited?

A. I would like every essay to be copy edited, at the very least, to get spelling checked and punctuation up to the standards of the printed word as opposed to the rather informal standard of personal blogs, however, if an author vehemently requests that their essay be printed exactly as it appeared on the web, we will honor this request, even going so far as to use a particularly silly font for that one essay if that's what the author wants!

Depending on the essay I may ask the author to make changes that reflect the shift from web to print. For example, essays which rely heavily on some other essay available only on the web may need to be somewhat rewritten so that the other essay is not a prerequisite. An essay that starts out as a reply to another essay, where on the web a hyperlink was sufficient to "incorporate by reference" the original, will have to be changed a bit to give the book's reader everything they need to enjoy the essay. I haven't chosen essays which can't stand alone so this will not be a big problem.

In a few cases the author and I may agree that the essay deserves to be expanded or rewritten for the written format. For example some of the articles appeared on the web in very terse or informal format and the writer may wish to expand a bit. This will probably be OK as long as the essay doesn't lose whatever made it interesting in the first place.

Q. Will authors retain rights to their works?

A. Apress will ask each author for nonexclusive rights to publish the work worldwide and in translation. Being nonexclusive means you can and should keep your article on the web or publish it anywhere else you see fit. Beyond that IANAL (I Am Not A Lawyer) so authors should carefully review the legal agreement Apress sends them.

Q. Is there a prize?

A. For each essay published the author will receive a purely symbolic $200 prize and five free copies of the book when it comes out. Authors will be entitled to purchase an unlimited number of additional copies at 60% off the cover price directly from Apress.

We're also designing a little trophy logo icon thing that authors can display alongside the original article on their website.

Q. What happens after an essay is selected?

When an essay is chosen, I'll chat with each author to discuss if there are any major changes we agree should be made for the print version.

Authors will have a choice of what we call the "long workflow" or the "short workflow." If an author wants to have a hand in the copy editing of their work for print and wants to approve every step of the process from web to print, they should opt for the long workflow. If they are happy to let our copy editors fix up their spelling and punctuation and are willing to trust the result in exchange for less work, they should opt for the short workflow.

The long workflow:

A) Apress sends author contract
B) Author signs and returns contract
C) An Apress editor will convert the document from HTML to a Word document in official "Apress" format and using Apress styles. At this point NO CHANGES WILL BE MADE, not even obvious spelling corrections.
D) The document will be sent to the author for review.
E) Author makes any changes necessary, replacing hyperlinks with footnotes, etc. and sends it to Joel.
F) Joel adds the editor's footnotes and sends it back to Author to review and accept or decline his changes.
G) Author sends this "first draft" to Apress copy editor.
H) Simultaneously, Joel sends an introduction to the copy editor.
I) Apress copy editor reviews style, punctuation, spelling, etc. of the essay and sends to author.
J) Apress copy editor reviews style, punctuation, spelling, etc. of the intro and sends it to Joel.
K) Author reviews copy editors changes and changes back anything they don't like
L) Joel reviews copy editor's changes in intro and changes back anything he doesn't like
M) Author sends document back to copy editor, who finalizes the document
N) Joel sends intro back to copy editor, who finalizes the intro
O) Pages are laid out -- the intro is combined with the article for the first time
P) Laid out pages ("galley proofs") are sent to the author as PDFs for final review
Q) Author sends them back with any emergency changes noted. At this point it is too late to change the pagination and Author will not be permitted to change anything in Joel's introduction.
R) Apress finalizes and publishes the book.

The short workflow:

A) Apress sends author contract
B) author signs and returns contract
C) An Apress editor will convert the document from HTML to a Word document in official "Apress" format and using Apress styles. At this point NO CHANGES WILL BE MADE, not even obvious spelling corrections.
F) Joel adds the editor's footnotes and introduction and sends it back to Apress.
I) Apress copy editor reviews style, punctuation, spelling, etc. of the intro and essay and sends it to Joel
L) Joel reviews copy editor's changes in intro and changes back anything he doesn't like
M) Joel sends document back to copy editor, who finalizes the document
O) Pages are laid out -- the intro is combined with the article
R) Apress finalizes and publishes the book.


The Guerrilla Guide to Interviewing

Thursday, March 23, 2000

Hiring the right people is extremely crucial to Fog Creek Software. In our field, there are three types of people. At one end of the scale, there are the unwashed masses, lacking even the most basic skills for this job. They are easy to ferret out and eliminate, often just by reviewing a resume and asking two or three quick questions. At the other extreme, are the brilliant superstars who write lisp compilers for fun, in a weekend, in Assembler for the Palm Pilot. And in the middle, you have a large number of "maybes" who seem like they might just be able to contribute something. The trick is telling the difference between the superstars and the maybes, because at Fog Creek Software we only hire the superstars. Here are some techniques for doing that.

First of all, the #1 cardinal criteria for getting hired at Fog Creek:

Smart, and
Gets Things Done.

That's it. That's all we're looking for. Memorize that. Recite it to yourself before you go to bed every night. Our goal is to hire people with aptitude, not a particular skill set. Any skill set that people can bring to the job will be technologically obsolete in a couple of years, anyway, so it's better to hire people that are going to be able to learn any new technology rather than people who happen to know SQL programming right this minute.

Smart is hard to define, but as we look at some possible interview questions we'll see how you can ferret it out. Gets Things Done is crucial. People who are Smart but don't Get Things Done often have PhDs and work in big companies where nobody listens to them because they are completely impractical. They would rather mull over something academic about a problem rather than ship on time. These kind of people can be identified because they love to point out the theoretical similarity between two widely divergent concepts. For example, they will say "Spreadsheets are really just a special case of programming language" and then go off for a week and write a thrilling, brilliant white paper about the theoretical computational linguistic attributes of a spreadsheet as a programming language. Smart, but not useful.

Now, people who Get Things Done but are not Smart will do stupid things, seemingly without thinking about them, and somebody else will have to come clean up their mess later. This makes them liabilities to the company because not only don't they contribute, but they soak up good people's time. They are the kind of people who copy big chunks of code around rather than writing a subroutine, because it gets the job done, just not in the smartest way.

The most important rule about interviewing:

Make A Decision

At the conclusion of the interview, you have to be ready to make a sharp decision about the candidate. There are only two possible outcomes to this decision: Hire or No Hire. Turn to your computer and send immediate feedback to the recruiter. The subject line should be the name of the candidate. The first line of the email should be Hire or No Hire. Then you should spend about 2 paragraphs backing up your decision.

There is no other possible answer. Never say, "Hire, but not in my group." This is rude and implies that the candidate is not smart enough to work with you, but maybe he's smart enough for those losers over in that other group. If you find yourself tempted to say "Hire, but not in my group," simply translate that mechanically to "No Hire" and you'll be OK. Even if you have a candidate that would be brilliant at doing 1 particular thing, but wouldn't be very good in another group, that's a No Hire. Things change so often and so rapidly that we need people that can succeed anywhere. If for some reason you find an idiot savant that is really, really, really good at SQL but completely incapable of ever learning any other topic, No Hire. They don't have a future at Fog Creek.

Never say "Maybe, I can't tell." If you can't tell, that means No Hire. It's really easier than you'd think. Can't tell? Just say no! Similarly, if you are on the fence, that means No Hire. Never say, "Well, Hire, I guess, but I'm a little bit concerned about..." That's a No Hire as well.

An important thing to remember about interviewing is this: it is much better to reject a good candidate than to accept a bad candidate. A bad candidate will cost a lot of money and effort and waste other people's time fixing all their bugs. If you have any doubts whatsoever, No Hire.

While you are conducting the interview, don't worry that if you reject a lot of people, Fog Creek won't be able to find anyone to hire. That's not your problem. It's the recruiter's problem, it's H.R.'s problem, it's Joel's problem, but it's not your problem. Keep asking yourself which is worse - that we grow into a big, lousy software company with lots of coconuts, or that we stay small but high quality? Of course, it's important to seek out good candidates and everybody should see it as a part of their mission to find and recruit smart people who get things done. But once you're actually interviewing someone, pretend that Fog Creek has plenty of great candidates. Never lower your standards no matter how hard it seems to find great candidates.

But how do you make this difficult decision? You just have to keep asking yourself during the interview: is this person smart? Does this person get things done? In order to be able to tell, you're going to have to ask the right questions.

Just for fun, here is the worst interview question on Earth: "What's the difference between varchar and varchar2 in Oracle 8i?" This is a terrible question. There is no possible, imaginable correlation between people that know that particular piece of useless trivia and people that Fog Creek wants to hire. Who cares what the difference is? You can find out online in about 15 seconds!

Actually, there are some even worse questions. I'll get to that later.

So now we get to the fun part: interview questions. My list of interview questions comes from my first job at Microsoft. There are actually hundreds of famous Microsoft interview questions. Everybody has a set of questions that they really like. You, too, will develop a particular set of questions and a personal interviewing style which helps you make the Hire/No Hire decision. Here are some techniques that I have used that have been successful.

Before the interview, I read over the candidates resume and jot down an interview plan on a scrap of paper. That's just a list of questions that I want to ask. Here's a typical plan for interviewing a programmer:

    1. Introduction
    2. Question about recent project candidate worked on
    3. Impossible Question
    4. C Function
    5. Are you satisfied?
    6. Design Question
    7. The Challenge
    8. Do you have any questions?

Before the interview, I am very, very careful to avoid anything that might give me some preconceived notions about the candidate. If you think that someone is smart before they even walk into the room, just because they have a Ph.D. from MIT, then nothing they can say in 1 hour is going to overcome that initial prejudice. If you think they are a bozo, nothing they can say will overcome that initial impression. An interview is like a very, very delicate scale -- it's very hard to judge someone based on a 1 hour interview and it may seem like a very close call. But if you know a little bit about the candidate beforehand, it's like a big weight on one side of the scale, and the interview is useless. Once, right before an interview, a recruiter came into my office. "You're going to love this guy," she said. BOY did this make me mad. What I should have said was, "well, if you're so sure I'm going to love him, why don't you just hire him instead of wasting my time going through this interview." But I was young and naïve, so I interviewed him. When he said not-so-smart things, I thought to myself, "gee, must be the exception that proves the rule." I looked at everything he said through rose-colored glasses. I wound up saying Hire even though he was a crappy candidate. You know what? Everybody else who interviewed him said No Hire. So: don't listen to recruiters; don't ask around about the person before you interview them; and never, ever talk to the other interviewers about the candidate until you've both made your decisions independently. It's the scientific method!

The Introduction phase of the interview is intended to put the candidate at ease. I spend about 30 seconds telling the person who I am and how the interview will work. I always reassure the candidate that we are interested in how he goes about solving problems, not the actual answer. By the way, in doing the interview, you should make sure that you are not sitting across a desk from the candidate. This creates a formal barrier which will not place the candidate at ease. It is better to move the desk against a wall, or to go around and sit on the other side of the desk with the candidate; this does help put the candidate at ease. This results in a better interview because it is less distorted by nervousness.

Part 2 is a question about some recent project that the candidate worked on. For interviewing college kids, ask them about their senior thesis, if they had one, or about a course they took that involved a long project that they really enjoyed. For example, sometimes I will ask, "what class did you take last semester that you liked the most? It doesn't have to be computer-related." Actually I am usually pretty happy if they choose a non-computer related course. Sometimes you look at their schedule, and it looks like they are taking the bare minimum number of Comp Sci courses, but every elective is something related to Music. Then they will tell you that their favorite course was Object Oriented Databases. Yeah, right. I'd be happier if they admitted that they just liked music more than computers, instead of sucking up.

When interviewing experienced candidates, you can talk about their previous job.

In this question, I'm looking for one thing: passion. When you find a project that the person worked on recently, these are all good signs:

  • They get very excited talking about it; they tend to talk more quickly and get animated. This shows that when they are interesting in something, they will be passionate about it. There are far too many people around that can work on something and not really care one way or the other. Even if they are passionately negative, this can be just as good a sign. "I was working on installing Foo Bar Mark II for my previous employer, but he was such a dope!" These are good candidates that we want to hire. Bad candidates just don't care and will not get enthusiastic at all during the interview. A really good sign that a candidate is passionate about something is that when they are talking about it, they will forget for a moment that they are in an interview. Sometimes a candidate comes in who is very nervous about being in an interview situation -- this is normal so I always overlook that. But then when you get them talking about Computational Monochromatic Art they will get extremely excited and lose all signs of nervousness. Good. I like passionate people who really care. (To see an example of Computational Monochromatic Art try unplugging your monitor.)
  • They are careful to explain things. I have rejected candidates because when they talked about their previous project, they couldn't explain it in terms that a normal person could understand. Often engineering majors will just assume that everyone knows what Bates Theorem is or what Peano's Axioms are. If they start doing this, stop them for a minute and say, "could you do me a favor, just for the sake of the exercise, could you please explain this in terms my grandmother could understand." At this point many people will still continue to use jargon and will completely fail to make themselves understood. GONG!
  • If the project was a team project, look for signs that they took a leadership role. A candidate might say: "we were working on X, but the boss said Y and the client said Z." I'll ask, "so what did you do?" A good answer to this might be "I got together with the other members of the team and wrote a proposal..." A bad answer might be, "Well, there was nothing I could do. It was an impossible situation." Remember, Smart and Gets Things Done. A good way to tell if somebody Gets Things Done is to see if historically they have tended to get things done in the past. In fact, you can even ask them directly to give you an example from their recent past when they took a leadership role and got something done -- overcame some institutional inertia, for example.

OK, the third thing on that list is the impossible question. This is fun. The idea is to ask a question that they have no possible way of answering, just to see how they handle it. "How many optometrists are there in Seattle?" "How many tons does the Washington Monument weigh?" "How many gas stations are in Los Angeles?" "How many piano tuners are there in New York?"

  • Smart candidates will realize that you are not quizzing them on their knowledge, and they will enthusiastically leap into trying to figure out some back-of-the-envelope answer. "Well, lets see, the population of LA is about 7 million; each person in LA has about 2.5 cars..." Of course it's OK if they are radically wrong. The important thing is that they leapt into the question enthusiastically. They may try to figure out the capacity of a gas station. "Gee, it takes 4 minutes to tank up, gas stations have about 10 pumps and are open about 18 hours a day..." They may try to figure it out by area. Sometimes they will surprise you with their creativity or ask for a Los Angeles yellow pages. All good signs.
  • Not-so-smart candidates will get flustered and upset. They will just stare at you like you landed from Mars. You have to coach them. "Well, if you were building a new city the size of Los Angeles, how many gas stations would you put in it?" You can give them little hints. "How long does it take to fill up a tank of gas?" Still, with not-smart candidates, you will have to drag them along while they sit there stupidly and wait for you to rescue them. These people are not problem solvers and we don't want them working for us.

For programming questions, I ask candidates to write a small function in C. Here are some typical problems I would ask:

  1. Reverse a string in place
  2. Reverse a linked list
  3. Count all the bits that are on in a byte
  4. Binary search
  5. Find the longest run in a string
  6. atoi
  7. itoa (great, because they have to use a stack or strrev)

You don't want to give them any problems that take more than about 5 lines of code; you won't have time for that.

Let's look at a couple of these in detail. #1: reverse a string in place. Every candidate I've ever interviewed in my life has done this wrong the first time. Without exception, they try to allocate another buffer and reverse the string into that buffer. The trouble is, who allocates the buffer? Who frees the buffer? In giving this question to dozens of candidates I found out an interesting fact. Most people who think that they know C really do not understand memory or pointers. They just don't get it. It's amazing that these people are working as programmers, but they are. With this question, here are some ways to judge the candidate:

  • Is their function fast? Look at how many times they call strlen. I've seen O(n^2) algorithms for strrev when it should be O(n), because they are calling strlen again and again in a loop.
  • Do they use pointer arithmetic? This is a good sign. Many "C programmers" just don't know how to make pointer arithmetic work. Now, ordinarily, I wouldn't reject a candidate just because he lacked a particular skill. However, I've discovered that understanding pointers in C is not a skill, it's an aptitude. In Freshman year CompSci, there are always about 200 kids at the beginning of the semester, all of whom wrote complex adventure games in BASIC for their Atari 800s when they were 4 years old. They are having a good ol'; time learning Pascal in college, until one day their professor introduces pointers, and suddenly, they don't get it. They just don't understand anything any more. 90% of the class goes off and becomes PoliSci majors, then they tell their friends that there weren't enough good looking members of the appropriate sex in their CompSci classes, that's why they switched. For some reason most people seem to be born without the part of the brain that understands pointers. This is an aptitude thing, not a skill thing ?euro;“ it requires a complex form of doubly-indirected thinking that some people just can't do.

For #3, you can see how well they learned the bitwise operators in C.... but this is a skill, not an aptitude, so you can help them with these. The interesting thing is to watch them write a subroutine that counts all the bits in a byte, then ask them to make it much, much faster. Really smart candidates will create a lookup table (after all, it's only got 256 entries) that they only have to create once. With good candidates, you can have a really interesting conversation about the different space/speed tradeoffs. Press them further: tell them you don't want to spend any time building the lookup table during initialization. Brilliant candidates might even suggest a caching scheme where bits are counted the first time they are used, and then stored in a lookup table so they don't have to be counted if they are used again. Really, really brilliant candidates will try to devise a way to compute the table using some kind of a shortcut taking advantage of the patterns that occur.

When you watch somebody write code, here are some techniques that may be helpful:

  • Always reassure them that you understand that it's hard to write code without an editor, and you will forgive them if their paper gets really messy. Also you understand that it's hard to write bug-free code without a compiler, and you will take that into account.
  • Some signs of a good programmer: good programmers have a habit of writing their { and then skipping down to the bottom of the page and writing their }s right away, then filling in the blank later. They also tend to have some kind of a variable naming convention, primitive though it may be... Good programmers tend to use really short variable names for loop indices. If they name their loop index CurrentPagePositionLoopCounter it is sure sign that they have not written a lot of code in their life. Occasionally, you will see a C programmer write something like if (0==strlen(x)), putting the constant on the left hand side of the == . This is a really good sign. It means that they were stung once too many times by confusing = and == and have forced themselves to learn a new habit to avoid that trap.
  • Good programmers plan before they write code, especially when there are pointers involved. For example, if you ask them to reverse a linked list, good candidates will always make a little drawing on the side and draw all the pointers and where they go. They have to. It is humanly impossible to write code to reverse a linked list without drawing little boxes with arrows between them. Bad programmers will start writing code right away.

Inevitably, you will see a bug in their function. So we come to question 5: Are you satisfied with that code? You may want to ask, "OK, so where's the bug?" The quintessential Open Ended Question From Hell. All programmers make mistakes, there's nothing wrong with that, they just have to be able to find them. With the string functions, they'll almost always forget to null-terminate the new string. With almost any function, they are likely to have off-by-one errors. They will forget semicolons sometimes. Their function won't work correctly on 0 length strings, or it will GPF if malloc fails... Very, very rarely, you will find a candidate that doesn't have any bugs the first time. In this case, this question is even more fun. When you say, "There's a bug in that code," they will review their code carefully, and then you get to see if they can be diplomatic yet firm in asserting that the code is perfect... In general, it's always a good idea to ask the candidate if they are satisfied with their answer before moving on. Be Regis.

Part 6: the design question. Ask the candidate to design something. Jabe Blumenthal, the original designer of Excel, liked to ask candidates to design a house. According to Jabe, he's had candidates who would go up to the whiteboard and immediately draw a square. A square! These were immediate No Hires. In design questions, what are you looking for?

  • Good candidates will try to get more information out of you about the problem. Who is the house for? As a policy, I will not hire someone who leaps into the design without asking more about who it's for. Often I am so annoyed that I will give them a hard time by interrupting them in the middle and saying, "actually, you forgot to ask this, but this is a house for a family of 48-foot tall blind giraffes."
  • Not-so-smart candidates think that design is like painting: you get a blank slate, and you can do whatever you want. Smart candidates understand that design is a difficult series of trade-offs. A great design question: design a trash can for a city street corner. Think of all the trade offs! It has to be easy to empty, but impossible to steal; it has to be easy to put things into, but hard for things to fly out of on a windy day; it has to be solid, yet inexpensive; in some cities, it has to be specially designed so that terrorists can't hide a bomb in it.
  • Creative candidates will often surprise you with an interesting, non-obvious answer. One of my favorite questions is Design a Spice Rack for Blind People. Inevitably, candidates will put Braille somewhere on the spice bottles, and it usually winds up being on top of the lid for various reasons which you'll discover after you've asked this question 100 times. I had one candidate who decided that it would be better to put the spices in a drawer, because it is more comfortable to scan Braille with your fingertips horizontal than vertical. (Try it!) This was so creative it surprised me -- in dozens of interviews, I had never heard that answer. And it really took a major creative "leap" outside of the bounds of the problem. On the strength of that answer alone, and no negatives, I hired the candidate, who went on to be one of the best program managers on the Excel team.
  • Look for closure. This is part of Get Things Done. Sometimes candidates will drift back and forth, unable to make a decision, or they will try to avoid hard questions. Sometimes they will leave difficult decisions unanswered and try to move on. Not good. Good candidates have a tendency to try to naturally keep things moving forward, even when you try to hold them back. If the conversation ever starts going around in circles, and the candidate says something like "well, we can talk about this all day, but we've got to do something, so let's go with decision X" that's a really good sign.

Which brings us to #7, The Challenge. This is fun. Throughout the interview, you look for the candidate to say something that is absolutely, positively, unarguably correct. Then you say, "wait a minute, wait a minute," and spend about 2 minutes playing devil's advocate. Argue with them when you are sure they are right.

  • Weak candidates will give in. No Hire.
  • Strong candidates will find a way to persuade you. They will have a whole laundry list of Dale Carnegie techniques to win you over. "Perhaps I'm misunderstanding you," they will say. But they will stand their ground. Hire.

Admittedly, in an interview situation, you are not equal parties. Thus there is a risk that the candidate will be afraid to argue with you because you are in a position of power over him. BUT, good candidates will tend to get fairly passionate about the argument, and they may momentarily forget that they are in an interview, and they will get very involved in trying to convince you. These are the people we want to hire.

Finally, you should ask the candidate if they have any questions. Some people like to look to see if the candidate will ask intelligent questions, which is a standard technique in the interviewing books. Personally, I don't care what questions they ask; by this point I've already made my decision. The trouble is, candidates have to see about 5-6 people in one day, and it's hard for them to ask 5-6 people different, brilliant questions, so if they don't have any questions, fine.

I always, always leave about 5 minutes a the end of the interview to sell Fog Creek. This is very important even if you are not going to hire them. If you've been lucky enough to find a really good candidate, you want to do everything you can at this point to make sure that they want to come to Fog Creek. Even if they are a bad candidate, you want to get them excited about Fog Creek Software so that they go away with a positive impression of the company. Think of it this way: these people are not just potential hires; they are also customers. They are also salesmen for our recruiting effort: if they think that Fog Creek is a great place to work, they will encourage their friends to apply.

Ah, I just remembered that I promised to give you some more examples of really bad questions to avoid.

First of all, avoid the illegal questions. Anything related to race, religion, gender, national origin, age, military service eligibility, veteran status, sexual orientation, or physical handicap is just illegal. If their resume says they were in the Army in 1990, don't ask them, even to make pleasant conversation, if they were in the Gulf war. It's against the law. If their resume says that they attended the Technion in Haifa, don't ask them, even conversationally, if they are Israeli. It's against the law. There's a pretty good discussion of what's illegal here. (But the rest of the interview questions at that site are pretty stupid).

Next, avoid any questions which might make it seem like we care about, or are discriminating based on, things which we don't actually care about or discriminate based on. The best example of this I can think of is asking someone if they have kids or if they are married. This might give the false impression that we think that people with kids aren't going to devote enough time to their work or that they are going to run off and take maternity leave.

Finally, avoid brain teaser questions like the one where you have to arrange 6 equal length matches to make exactly 4 identical perfect triangles. If it's an "aha!" question, you don't get any information about "smart/get things done" by figuring out if they happen to make the mental leap or not.

Interviewing is more of an art than a science, but if you remember the Smart/Gets Thing Done principle you will be in good shape. When you get a chance, ask some of your co-workers what their favorite questions are and what kinds of answers they look for. In the Building 16 cafeteria in Redmond this is a perennial favorite topic of lunchtime conversation.


The Iceberg Secret, Revealed

Wednesday, February 13, 2002

"I don't know what's wrong with my development team," the CEO thinks to himself. "Things were going so well when we started this project. For the first couple of weeks, the team cranked like crazy and got a great prototype working. But since then, things seem to have slowed to a crawl. They're just not working hard any more." He chooses a Callaway Titanium Driver and sends the caddy to fetch an ice-cold lemonade. "Maybe if I fire a couple of laggards that'll light a fire under them!"

Meanwhile, of course, the development team has no idea that anything's wrong. In fact, nothing is wrong. They're right on schedule.

Don't let this happen to you! I'm going to let you in on a little secret about those non-technical management types that will make your life a million times easier. It's real simple. Once you know my secret, you'll never have trouble working with non-technical managers again (unless you get into an argument over the coefficient of restitution of their golf clubs).

It's pretty clear that programmers think in one language, and MBAs think in another. I've been thinking about the problem of communication in software management for a while, because it's pretty clear to me that the power and rewards accrue to those rare individuals who know how to translate between Programmerese and MBAese.

Since I started working in the software industry, almost all the software I've worked on has been what might be called "speculative" software. That is, the software is not being built for a particular customer -- it's being built in hopes that zillions of people will buy it. But many software developers don't have that luxury. They may be consultants developing a project for a single client, or they may be in-house programmers working on a complicated corporate whatsit for Accounting (or whatever it is you in-house programmers do; it's rather mysterious to me).

Have you ever noticed that on these custom projects, the single most common cause of overruns, failures, and general miserableness always boils down to, basically, "the (insert expletive here) customer didn't know what they wanted?"

Here are three versions of the same pathology:

  1. "The damn customer kept changing his mind. First he wanted Client/Server. Then he read about XML in Delta Airlines Inflight Magazine and decided he had to have XML. Now we're rewriting the thing to use fleets of small Lego Mindstorms Robots."
  2. "We built it exactly the way they wanted. The contract specified the whole thing down to the smallest detail. We delivered exactly what the contract said. But when we delivered it, they were crestfallen."
  3. "Our miserable sales person agreed to a fixed price contract to build what was basically unspecified, and the customer's lawyers were sharp enough to get a clause in the contract that they don't have to pay us until 'acceptance by customer,' so we had to put a team of nine developers on their project for two years and only got paid $800."

If there's one thing every junior consultant needs to have injected into their head with a heavy duty 2500 RPM DeWalt Drill, it's this: Customers Don't Know What They Want. Stop Expecting Customers to Know What They Want. It's just never going to happen. Get over it.

Instead, assume that you're going to have to build something anyway, and the customer is going to have to like it, but they're going to be a little bit surprised. YOU have to do the research. YOU have to figure out a design that solves the problem that the customer has in a pleasing way.

Put yourself in their shoes. Imagine that you've just made $100,000,000 selling your company to Yahoo!, and you've decided that it's about time to renovate your kitchen. So you hire an expert architect with instructions to make it "as cool as Will and Grace's Kitchen." You have no idea how to accomplish this. You don't know that you want a Viking stove and a Subzero refrigerator -- these are not words in your vocabulary. You want the architect to do something good, that's why you hired him.

The Extreme Programming folks say that the solution to this is to get the customer in the room and involve them in the design process every step of the way, as a member of the development team. This is, I think, a bit too "extreme." It's as if my architect made me show up while they were designing the kitchen and asked me to provide input on every little detail. It's boring for me, if I wanted to be an architect I would have become an architect.

Anyway, you don't really want a customer on your team, do you? The customer-nominee is just as likely to wind up being some poor dweeb from Accounts Payable who got sent to work with the programmers because he was the slowest worker over there and they would barely notice his absence. And you're just going to spend all your design time explaining things in words of one syllable.

Assume that your customers don't know what they want. Design it yourself, based on your understanding of the domain. If you need to spend some time learning about the domain or if you need a domain expert to help you, that's fine, but the design of the software is your job. If you do your domain homework and create a good UI, the customer will be pleased.

Now, I promised to tell you a secret about translating between the language of the customers (or nontechnical managers) of your software and the language of programmers.

You know how an iceberg is 90% underwater? Well, most software is like that too -- there's a pretty user interface that takes about 10% of the work, and then 90% of the programming work is under the covers. And if you take into account the fact that about half of your time is spent fixing bugs, the UI only takes 5% of the work. And if you limit yourself to the visual part of the UI, the pixels, what you would see in PowerPoint, now we're talking less than 1%.

That's not the secret. The secret is that People Who Aren't Programmers Do Not Understand This.

There are some very, very important corollaries to the Iceberg Secret.

Important Corollary One. If you show a nonprogrammer a screen which has a user interface that is 90% worse, they will think that the program is 90% worse.

I learned this lesson as a consultant, when I did a demo of a major web-based project for a client's executive team. The project was almost 100% code complete. We were still waiting for the graphic designer to choose fonts and colors and draw the cool 3-D tabs. In the meantime, we just used plain fonts and black and white, there was a bunch of ugly wasted space on the screen, basically it didn't look very good at all. But 100% of the functionality was there and was doing some pretty amazing stuff.

What happened during the demo? The clients spent the entire meeting griping about the graphical appearance of the screen. They weren't even talking about the UI. Just the graphical appearance. "It just doesn't look slick," complained their project manager. That's all they could think about. We couldn't get them to think about the actual functionality. Obviously fixing the graphic design took about one day. It was almost as if they thought they had hired painters.

Important Corollary Two. If you show a nonprogrammer a screen which has a user interface which is 100% beautiful, they will think the program is almost done.

People who aren't programmers are just looking at the screen and seeing some pixels. And if the pixels look like they make up a program which does something, they think "oh, gosh, how much harder could it be to make it actually work?"

The big risk here is that if you mock up the UI first, presumably so you can get some conversations going with the customer, then everybody's going to think you're almost done. And then when you spend the next year working "under the covers," so to speak, nobody will really see what you're doing and they'll think it's nothing.

Important Corollary Three. The dotcom that has the cool, polished looking web site and about four web pages will get a higher valuation than the highly functional dotcom with 3700 years of archives and a default grey background.

Oh, wait, dotcoms aren't worth anything any more. Never mind.

Important Corollary Four. When politics demands that various nontechnical managers or customers "sign off" on a project, give them several versions of the graphic design to choose from.

Vary the placement of some things, change the look and feel and fonts, move the logo and make it bigger or smaller. Let them feel important by giving them non-crucial lipstick-on-a-chicken stuff to muck around with. They can't do much damage to your schedule here. A good interior decorator is constantly bringing their client swatches and samples and stuff to choose from. But they would never discuss dishwasher placement with the client. It goes next to the sink, no matter what the client wants. There's no sense wasting time arguing about where the dishwasher goes, it has to go next to the sink, don't even bring it up; let the clients get their design kicks doing some harmless thing like changing their mind 200 times about whether to use Italian Granite or Mexican Tiles or Norwegian wood butcher-block for the countertops.

Important Corollary Five. When you're showing off, the only thing that matters is the screen shot. Make it 100% beautiful.

Don't, for a minute, think that you can get away with asking anybody to imagine how cool this would be. Don't think that they're looking at the functionality. They're not. They want to see pretty pixels.

Steve Jobs understands this. Oh boy does he understand this. Engineers at Apple have learned to do things that make for great screen shots, like the gorgeous new 1024x1024 icons in the dock, even if they waste valuable real estate. And the Linux desktop crowd goes crazy about semitransparent xterms, which make for good screenshots but are usually annoying to use. Every time Gnome or KDE announces a new release I go straight to the screenshots and say, "oh, they changed the planet from Jupiter to Saturn. Cool." Never mind what they really did.

Remember the CEO at the beginning of this article? He was unhappy because his team had showed him great PowerPoints at the beginning -- mockups, created in Photoshop, not even VB. And now that they're actually getting stuff done under the covers, it looks like they're not doing anything.

What can you do about this? Once you understand the Iceberg Secret, it's easy to work with it. Understand that any demos you do in a darkened room with a projector are going to be all about pixels. If you can, build your UI in such a way that unfinished parts look unfinished. For example, use scrawls for the icons on the toolbar until the functionality is there. As you're building your web service, you may want to consider actually leaving out features from the home page until those features are built. That way people can watch the home page go from 3 commands to 20 commands as more things get built.

More importantly, make sure you control what people think about the schedule. Provide a detailed schedule in Excel format. Every week, send out self-congratulatory email talking about how you've moved from 32% complete to 35% complete and are on track to ship on December 25th. Make sure that the actual facts dominate any thinking about whether the project is moving forward at the right speed. And don't let your boss use Callaway Titanium Drivers, I don't care how much you want him to win, the USGA has banned them and it's just not fair.

Discuss


The Joel Test: 12 Steps to Better Code

Wednesday, August 09, 2000

Have you ever heard of SEMA? It's a fairly esoteric system for measuring how good a software team is. No, wait! Don't follow that link! It will take you about six years just to understand that stuff. So I've come up with my own, highly irresponsible, sloppy test to rate the quality of a software team. The great part about it is that it takes about 3 minutes. With all the time you save, you can go to medical school.


The Law of Leaky Abstractions

Monday, November 11, 2002

There's a key piece of magic in the engineering of the Internet which you rely on every single day. It happens in the TCP protocol, one of the fundamental building blocks of the Internet.

TCP is a way to transmit data that is reliable. By this I mean: if you send a message over a network using TCP, it will arrive, and it won't be garbled or corrupted.

We use TCP for many things like fetching web pages and sending email. The reliability of TCP is why every exciting email from embezzling East Africans arrives in letter-perfect condition. O joy.

By comparison, there is another method of transmitting data called IP which is unreliable. Nobody promises that your data will arrive, and it might get messed up before it arrives. If you send a bunch of messages with IP, don't be surprised if only half of them arrive, and some of those are in a different order than the order in which they were sent, and some of them have been replaced by alternate messages, perhaps containing pictures of adorable baby orangutans, or more likely just a lot of unreadable garbage that looks like the subject line of Taiwanese spam.

Here's the magic part: TCP is built on top of IP. In other words, TCP is obliged to somehow send data reliably using only an unreliable tool.

To illustrate why this is magic, consider the following morally equivalent, though somewhat ludicrous, scenario from the real world.

Imagine that we had a way of sending actors from Broadway to Hollywood that involved putting them in cars and driving them across the country. Some of these cars crashed, killing the poor actors. Sometimes the actors got drunk on the way and shaved their heads or got nasal tattoos, thus becoming too ugly to work in Hollywood, and frequently the actors arrived in a different order than they had set out, because they all took different routes. Now imagine a new service called Hollywood Express, which delivered actors to Hollywood, guaranteeing that they would (a) arrive (b) in order (c) in perfect condition. The magic part is that Hollywood Express doesn't have any method of delivering the actors, other than the unreliable method of putting them in cars and driving them across the country. Hollywood Express works by checking that each actor arrives in perfect condition, and, if he doesn't, calling up the home office and requesting that the actor's identical twin be sent instead. If the actors arrive in the wrong order Hollywood Express rearranges them. If a large UFO on its way to Area 51 crashes on the highway in Nevada, rendering it impassable, all the actors that went that way are rerouted via Arizona and Hollywood Express doesn't even tell the movie directors in California what happened. To them, it just looks like the actors are arriving a little bit more slowly than usual, and they never even hear about the UFO crash.

That is, approximately, the magic of TCP. It is what computer scientists like to call an abstraction: a simplification of something much more complicated that is going on under the covers. As it turns out, a lot of computer programming consists of building abstractions. What is a string library? It's a way to pretend that computers can manipulate strings just as easily as they can manipulate numbers. What is a file system? It's a way to pretend that a hard drive isn't really a bunch of spinning magnetic platters that can store bits at certain locations, but rather a hierarchical system of folders-within-folders containing individual files that in turn consist of one or more strings of bytes.

Back to TCP. Earlier for the sake of simplicity I told a little fib, and some of you have steam coming out of your ears by now because this fib is driving you crazy. I said that TCP guarantees that your message will arrive. It doesn't, actually. If your pet snake has chewed through the network cable leading to your computer, and no IP packets can get through, then TCP can't do anything about it and your message doesn't arrive. If you were curt with the system administrators in your company and they punished you by plugging you into an overloaded hub, only some of your IP packets will get through, and TCP will work, but everything will be really slow.

This is what I call a leaky abstraction. TCP attempts to provide a complete abstraction of an underlying unreliable network, but sometimes, the network leaks through the abstraction and you feel the things that the abstraction can't quite protect you from. This is but one example of what I've dubbed the Law of Leaky Abstractions:


The Project Aardvark Spec

Wednesday, August 17, 2005

As the final version of Copilot.com (what this spec calls ?euro;œAardvark?euro;?) went into production in early August, 2005, this spec is now of historical interest only. I?euro;™m making it available as a part of Joel on Software because many people have asked to see some samples of the kinds of specs we write at Fog Creek. (You may also be interested in my series on Painless Functional Specifications.)

There was one big mistake in the spec. When I wrote it I had assumed that port 443, used by the secure (SSL) http protocol, would be open even on networks that are configured to use proxy servers. I mistakenly believed that proxy servers could not forward SSL traffic because that would appear, to the browser, to be a ?euro;œman in the middle?euro;? attack. This, it turns out, was erroneous, false, and incorrect, and after shipping the first beta we discovered just how wrong it was. Thus, the final implementation of Copilot 1.0 required a lot of unplanned work to support proxy detection and different kinds of proxy servers. We got most of that work done in time for 1.0 and are now finishing off the rest of it for a release we?euro;™re calling 1.1.

That said, the extra proxy work only added about 10% to the overall development effort. The reason you write a spec is not to solve every possible problem in advance: the reason you write a spec is to solve as many problems as you possibly can in advance so that you minimize the number of surprises that come up during development. In our case, since we were working with summer interns who literally had to walk out the door and go back to school, we literally could not afford to underestimate the amount of work that was needed to build Project Aardvark or I would be stuck debugging the thing deep into September while the kids drank beer and played poker, and we couldn?euro;™t have that.

Many times, thinking things out in advance saved us serious development headaches later on. When I wrote the first draft of this spec, I had a more complicated flowchart that assumed that either party (helper or victim) could initiate the process. This became extremely complicated and convoluted for various reasons. As I worked through the screens that would be needed to allow either party to initiate the process, I realized that Aardvark would be just as useful, and radically simpler, if the helper was required to start the whole process. Making this change in the spec took an hour or two. If we had made this change in code, it would have added weeks to the schedule. I can?euro;™t tell you how strongly I believe in Big Design Up Front, which the proponents of Extreme Programming consider anathema. I have consistently saved time and made better products by using BDUF and I?euro;™m proud to use it, no matter what the XP fanatics claim. They?euro;™re just wrong on this point and I can?euro;™t be any clearer than that.

There was one other significant change we made late in the process which is not reflected in the spec. After usability testing, we discovered that people were consistently confused by our policy of having the application delete itself. This was not what users expect. We modified the final product so that the helper could check a box to cause the victim?euro;™s executable to be deleted, but it wasn?euro;™t done by default. This way the helper, who is presumably more computer savvy, could insure that no unsafe executables were left around on the victim?euro;™s computer. I think this change was about 2 hours of work, so it didn?euro;™t impact the schedule.

Finally, you?euro;™ll notice a big ?euro;œcoding conventions?euro;? section in the middle of the spec. Frankly, this doesn?euro;™t belong in a functional specification, and it should really have been a separate document.

Still, as I reread this spec today, I?euro;™m surprised by what a good job the Aardvarkians did implementing everything. I seriously expected we would have to cut features; instead, we shipped with everything we wanted in version 1 and even did some of the features this spec refers to as version 2 features.

Download The Project Aardvark Spec in PDF Format (about 120KB). If you link to the spec, please link to this page, not the PDF file itself.


The Ricochet Wireless Modem (a Review)

Wednesday, December 20, 2000

Since my company is moving to a new office, I've been calling around trying to find an Internet provider. In Manhattan these days, you have a zillion choices. It seems like every Internet provider in the known universe offers service in Manhattan.

We're probably going to get a T1 line from Savvis. Getting a T1 installed takes a couple of months, and we were worrying about what to do while we wait. ISDN and DSL are nightmares to install and take just as long to get installed. In desperation I was considering a dedicated dial up service, which is usually expensive as hell (something like $100 - $150 per month) and slow (modem speed). Or maybe a bonded modem service, where you use two modems and some expensive equipment to get double modem speed (realistically, that would be about 80kbps).

Then I found

Anyway, I called Juno and ordered Juno Express Wireless. They will sell you a modem for $99, and the monthly service is $80 for unlimited usage. This is a fantastic deal if you've ever priced wireless Internet access in the past. (I remember when it was $25 per megabyte, not so long ago, for download speeds around 9.6kbps.)

The first Ricochet modems ran at 28.8 kbps. They are now transitioning over to a new technology that provides 128kbps, which is what I'm using. Wow! That's dual channel ISDN speed! The trick is that the high speeds are only available in a few places. Luckily, Manhattan is one. Metrocom has been installing small shoebox-sized transmitters on light poles throughout Manhattan, since the technology they are using doesn't have very long range (the cells aren't nearly as large or as strong as typical cell-phone cells).

The modem itself is the size of a small paperback book, with a little antenna that folds out. It connects to your laptop computer via a USB port. (It can use a serial port, but the speed is slower if you do that). It has its own battery that runs for 6 hours so it doesn't have to suck up your laptop's battery. Yes, it's really 6 hours... I ran it down today.

Installation was almost easy. Once I got everything set up, Juno was still giving me weird error messages. Luckily I remembered, from my time as a programmer at Juno, that the Juno system requires you to check your mail at least once when any features of your account change so that the Juno servers can update the client with your new service information. I think that if I didn't have this "inside information" I would have been pretty frustrated, and I would have had to call Juno tech support. Anyway, by checking my Juno email once using my home DSL connection, everything was fixed up and I was able to start using Ricochet successfully.

My apartment is in the back of a big, brick brownstone in a dense part of Manhattan's Upper West Side. So the signal wasn't very strong. The maps on Ricochet's site implied that they have more transmitters in midtown anyway. At home, the service was very bursty. When it worked, it was fast, but a lot of the time, it was super-slow (like 14.4 modem speed). This doesn't surprise me or bother me very much... I can barely get cellular reception from behind all the masonry on my block.

So I took the modem out to a local cafe on Broadway and 70th, near a major avenue, and tried it from there. Tada! This is how it was meant to work. This thing is fast. You do get 128 kbps, sometimes more. I tried a whole bunch of the various line-speed-tests around the Internet, like C|net's. These tests are shockingly unscientific, since there are so many other things that can slow down a connection between your machine and the server where they are running, but I did get results between 70 kbps and 180 kbps. Nice! The overall feel of the thing is great. While I've been writing this article, I've been uploading pictures, sending email, browsing the web, and some people have emailed me some really huge attachments which I've forwarded on... the overall speed is great, much much faster than a "56kbps" modem which usually gets you more like 40 kbps. And it's wireless! And it's easy to install! And it only took two days from placing an order to being online! (They must have shipped the modem the day I called).

I also tried the service from a friend's house around the corner; it works at around 128kbps there, too. Slick.

Now, $80 a month may sound like a lot for a home user. But for our small office of five people, for a temporary solution while we wait for the T1, it's going to be great. And for working as consultants, where we're all over the city working at clients' sites, working from cafes and libraries, etc., it will also be great; I'm thinking of equipping my entire team with these so we don't have to rely on clients' Internet connections.

There are two things I would change. First, I wish there was a capability to attach a big antenna when you want to have a fixed installation with better reception. Maybe that's an FCC restriction?

I also wish it gave you some idea of the signal strength as you moved the antenna around so you could adjust it for optimal reception. There is no way to tell which way to wiggle the antenna.

Note: Someone on the Yahoo message boards has since informed me that you can check the signal strength by pressing in the power slide switch, which gives you from 1 - 4 beeps indicating signal strength. Totally non-obvious and not in the documentation, but it seems to work!

Another big problem is the coverage. This thing works in the downtown areas of 10 cities. But in the past couple of years, the places where I want wireless access are not downtowns. Maybe I'm staying with a friend and don't want to tie up their modem. Or I'm at the beach. Last year I was in the middle of the Navajo reservation and I needed Internet access. I suspect Ricochet ain't never gonna have base stations in the middle of the Navajo reservation.

August 2, 2001: Ricochet Closes Down. They spent $500,000,000 and went out of business owing almost a billion dollars, with 51,000 subscribers. They would have had to make about $30,000 per subscriber for the business idea to have worked.


The Road to FogBugz 4.0: Part I

Monday, March 28, 2005

It started with a phone call from a customer, back in the old office.

Most of our customers, thankfully, understand that the idea here at Fog Creek is to make shrinkwrapped, off-the-shelf software that you buy for a low price and it already does what you need, and if it doesn't, well, we'd like to hear about that, but for $99 you're not getting a customized version of the software, sorry.

Some of our customers still think we're one of those big enterprise software companies where you call them up, negotiate with a salesperson for three months, and, on the last day of the fiscal quarter, force the salesperson to promise a long list of new features in exchange for a half-million dollar contract.

That's nice, but we don't have salespeople and we're not one of those vendors. Our customers are happy that our price point is low but not all of them quite understand the implications. They ask us to fly out to their headquarters to give a demo of the software to their development team. They send us long spreadsheets with lists of features and ask us to check off the features we support. They even send us RFPs (shudder). RFP stands for "Request for Proposal." It's a request by a large company for a custom proposal from a small company. The small company works on the 200 page laser-printed proposal like mad for three weeks and Fedexes it in great expense and at the last minute, where it gets put in the trash because the large company has their favorite vendor who takes them on a helicopter to Atlantic City on junkets involving blackjack and strippers, and who is going to get the contract no matter what, but someone in purchasing for some unexplained reason, maybe he's bucking for a promotion is insisting that the proposal be opened up to "competitive bidding" and the small company has been chosen as a victim to write up a proposal that has no chance of being accepted just to make the process look a little bit less corrupt, and if you're a small company, I would recommend that you don't fall for it and don't spend any time responding to RFPs unless it's already understood that you're going to get the contract.

ANYHOO.

The good thing about such customers is that they give us useful feedback about what features we should be adding to FogBugz. One of our mantras here at Fog Creek is "Listen to your customers, not your competitors." So when we get calls from potential big customers we listen.

Back then, early in the planning stages of FogBugz 4.0, the call was from a large not-for-profit organization. They were considering using so many copies of FogBugz that they needed a way for department managers to see their own departments' cases in one place, even when each department managed multiple projects.

The short term solution would have been for us to add a table of departments and make a report based on departments. But we're an off-the-shelf company, and when we do something, we want to do it in a general way that's useful to a lot of customers. So we took a little bit more time and designed a more general feature and solved a large class of customer problems with one feature. In addition to customers who wanted things sorted out by department, we had customers who were typical consulting shops and wanted to sort things out by client. Most of these customers wanted privacy features so their clients could access FogBugz without seeing any

So, OK, it took a little bit of time to deliver this feature to this particular customer. From that phone call to the time we delivered an alpha release to the customer with the new feature about twelve months elapsed. That's more because we're still oriented around big major releases every year and a half. I justified the reason for this schedule in the article Picking a Ship Date.

Many of the other features in FogBugz 4.0, which we finally shipped on February 22, were based on customer feedback, too. We kept hearing customers asking how to attach screenshots to a bug. "Oh, it's easy," we said. "Alt+Print Screen, run paintbrush, paste, save as a file somewhere, run your browser, go to the FogBugz homepage, hit "New Case," describe the case, click the file browse button, and find the file you just created." What could be easier?

Maybe it could be a bit easier.

Finally I thought "how hard could it be to make a little taskbar lint icon thingy that grabs a screenshot and makes a bug out of it?" Not so hard.

I got the Windows version done in The Proverbial Weekend (defined as one weekend to get it working, two weeks to fix bugs, another week to rewrite it to workaround new bugs introduced by a patch to Internet Explorer). Daniel Berlinger knocked out the Macintosh version in REALbasic in a couple of weeks. And now entering a bug that you see on screen is a matter of two clicks.

It's great.

I've found that about 30% of the bugs I enter can be completely described with a screenshot. Here's a complete bug report I entered on Monday:

The red rectangle comes from a highlight tool built into the screenshot program. The "wha?" I typed was probably overkill. If I had been willing to live with an untitled bug this entire bug report could have been entered with four clicks and a drag (that sounds like a good name for a band: "Four Clicks and a Drag." Or the Fog Creek development team, now that I think of it.)

The thing is, as far as I know, not that I pay close attention, none of our competitors has this feature.

If one of our competitors think this is cool, they can copy us, but it'll take them a while, especially if they read my site and bought my line about only shipping every 18 months.

No amount of listening to our competitors would have motivated us to do a screenshot feature. Our customers didn't think to ask for it, but we did notice that they kept asking for ways to attach screenshots, so that's what we did.

Come back tomorrow for Part II, in which I talk about dog food.


The Road to FogBugz 4.0: Part II

Tuesday, March 29, 2005

A long time ago I wrote an article called What is the Work of Dogs in this Country? about the benefits of eating your own dogfood, a quaint expression in the software industry that means "using your own product."

We're totally devoted to the idea of using our own software as much as possible... all my writing is done inside CityDesk (a very unstable developer build, at that, which drives everyone completely crazy), all our development is planned and tracked in FogBugz, of course, and even our customer service operation is entirely supported and tracked with FogBugz, which is why when you email us you always get a reply.

Well, almost always.

About a year ago, I noticed that people were emailing us, sometimes, and not getting responses.

This was not good.

I dug up their messages in FogBugz.

RESOLVED: SPAM, it said. Someone handling the customer service queue had decided that the email was spam and closed it without responding.

When I looked closer, it was usually because the subject line sounded kinda spammy or was blank, but sometimes it was just the proverbial Operator Error.

In the meantime I had been using SpamBayes for my own personal email. This is probably the best implementation of what is probably the best spam filtering algorithm out there: Bayesian filtering, invented by Paul Graham and first published in August 2002 in his seminal article A Plan for Spam. Bayesian filtering learns from experience. When it makes a mistake trying to classify an email as spam or non-spam, you correct it, and it looks for clues in that message so it can do a better job at classification in the future.

And I thought, gosh, here I am paying a college graduate to go through email and decide whether or not it's spam, and he's flagging 1% of email incorrectly, when there's this algorithm called Bayesian filtering which in my experience has far fewer false positives (something like 0.01% false positives). So while everybody is worried about how spam filters might inadvertently delete the proverbial crucial email from a customer, in practice, in the presence of lots of spam, human beings are far more likely to delete a real email than a well-implemented Bayesian filter.

The other thing about the Bayesian algorithm, by the way, is that it has to be implemented in the email client, not the server, because it needs to be taught about what one particular person's legitimate email looks like. For example when we get email at Fog Creek, the word "FogBugz" in the body of the email is strong evidence that the email is not spam while the word "mortgage" in the body of the email is strong evidence that the email is spam. However if you're a real estate agent or bank, the word mortgage probably occurs all the time in legitimate email. So basically Bayesian filtering is not going to work unless it knows what messages you deleted as spam and which suspicious messages you recovered. The Ipswitch guys who make a mail server tried to implement Bayesian filtering on the server, because that's what they have, and frankly that system just doesn't work.

Since spam filtering needs to be done on the client, and FogBugz is an email client, we decided that spam filtering needs to be done in FogBugz.

Now, all that probability stuff is too hard for me, so we asked Summer Intern Ben Kamens to figure it out. In a matter of weeks he had a beautiful, speedy implementation of Bayesian filtering up and running in FogBugz.

Bayesian filtering is really just a specialization of a long established AI algorithm for sorting documents based on training, and we thought, gosh, why not generalize the algorithm so that in addition to sorting incoming email into "spam," "suspect," and "ham," it also sorted all the ham into piles, for example, "sales," "tech support," and "job applications"?

Ben spent a few more weeks studying this and came up with the idea of using a tournament algorithm. He's a college athlete. They think of the world in terms of tournaments. It worked quite well. With 24 hours of training I had it sorting my own inbox into personal email, Fog Creek email, Joel on Software email, Joel on Software translator coordination email, Subversion checkin comments, bug notifications, and sales reports. Frankly it did an astonishingly good job with 100% accuracy on the obvious emails, like the checkin comments that all had the same structure, and well over 90% accuracy on the hard ones, like telling personal email from Joel on Software reader mail.

I was tempted to try and get it to sort email into messages from Republicans vs. messages from Democrats, but that was pushing my luck.

By the way, the paper Ben wrote at the end of the summer describing his algorithm is here (PDF). This paper is the ultimate rebuttal to those grumpy people who email me, barely able to conceal their disgust, saying, "why do you need to hire such smart people to work on bug tracking software?"

The cool thing about the implementation in FogBugz is that if the sorter makes a mistake, it's OK. You just correct it and get on with your life, and it learns from its mistake. But in the meantime it saves you 95% of the work directing incoming email to the right people.

Snippets

We got one other good idea for FogBugz 4.0 from dogfooding it.

As long as FogBugz has had the ability to receive mail, all customer service email sent to Fog Creek goes into our own FogBugz database where anyone can answer it.

Over time I've come to notice the occasional customer who thought we were being rude to him in an email message. On closer examination, we weren't being rude, but the email we sent seemed rude, and it was usually because it was incredibly terse and to-the-point. We decided that instead of sending emails like:

Yep, this is fixed in the latest version.

... we should be sending emails like:

Hi! Thanks for writing to us. I think I know what you're talking about, and it's definitely a bug in our product. The good news is: it's fixed! You can download the latest version by logging onto our online store at https://shop.fogcreek.com with your order ID number and email address, and that should solve this problem once and for all. If you don't have your order ID number just let me know and I'll be happy to look it up for you, or call us here at the Fog Creek office at 866-FOG-CREEK.

Please let me know if there's any other way I can be of assistance!

All the best,

(signature of a real human being)

It's sort of like translating from English to Japanese. I have never been to Japan but my father, a linguist, once told me the story of the train station in Tokyo, where the announcements were made in Japanese and English. You would hear four or five minutes of nonstop Japanese and then the English translation would be "The train to Osaka is on platform 4." It seems that in Japanese there is simply no way to say something that simple without cosseting it heavily in a bunch of formal etiquette-stuff. And it turns out the same thing applies to email messages, even in English. The moral of the story is that given two email messages with the same semantic content, the terse one is more likely to come across sounding rude. But given the amount of email correspondence we have to deal with here, we don't have time to be Emerson on every customer support email.

Thus was born the idea of snippets: canned bits 'n' pieces of email that you could insert into an outgoing reply with a few keystrokes. The theory is that you define a snippet like "Please let me know if there's any other way I can be of assistance" and then assign it to a short code like "2". Now while your composing your reply, you just hit 2 and then press the backquote key ` and (ding!) it is instantly replaced with the long text it represents right in front of your eyes. It's really cool and it saves a ton of time and allows us to produce the verbose email replies that are less likely to be misinterpreted. We have built up an extensive library of snippets for all kinds of "faqs" and common parts of email messages.

In tomorrow's installment of The Road to FogBugz, a look at Thistle, our proprietary ASP to PHP compiler/translator.


The Road to FogBugz 4.0: Part III

Wednesday, March 30, 2005

One day in the summer of 2003, after my friend Uday had finally told me for about the 17th time that he would have bought FogBugz but they didn't have any Windows servers to run it on, I decided to figure out a plan for getting a Unix version. We had looked into Chilisoft, an implementation of ASP and VBScript that runs on Unix, but we didn't like that idea much because at the time it cost something like $1000 per server and that wouldn't be acceptable to our customers, who could have bought a whole 'nother Windows server for not much more than that.

One morning walking to work I thought, gosh, it would be cool to have a summer intern, and I could tell the intern to write an ASP compiler that generated PHP as its output. Then we could produce a Unix version of FogBugz from our source code automatically, and as we added new features to the ASP version, they would magically show up in the PHP version without any more work.

Lo and behold that day this kid named Jimmy calls up the office. He's just back from Junior year in Italy and he needed a summer job. When I gave him one of my (unpublished!) interview questions (don't look for it online, you won't find it) he answered it in exactly the same way as I had answered the same question when Brian MacDonald asked it to me for an internship at Microsoft back in 1990. All the same false starts, the same initially-wrong answer, and eventually, the ideal answer, Jimmy was sounding like a clone of me.

Against my better judgment, knowing myself, we hired him for the summer, and thus was born the Fog Creek Summer Internship program.

Jimmy didn't know PHP, or ASP, or Java, really, but we thought Java was a good implementation language for an ASP to PHP compiler, so he dealt. Within a few weeks we had something up and running. By the end of the summer it was pretty decent, running FogBugz like a champ, although after Jimmy went back to school, getting all the tiny details right took Michael another couple of months.

I keep calling this thing an ASP to PHP compiler, which leads many people to email me saying "don't you mean translator?"

Let me explain.

In computer science jargon a translator IS a compiler. It's exactly the same thing. Those are synonyms.

In common usage, people think of a compiler as generating machine code, but it doesn't have to. Java and .NET compilers generate bytecode, for example. And Stroustrup's first C++ compiler, called Cfront, actually generated C code because C was portable and this made it possible to run C++ on any system that had a C compiler. When you're inventing a new language it's a waste of time generating machine language output for every architecture when you can just generate nice messy C code and let cc generate the machine language and do all the back end and optimization stuff. Cfront was terribly confusing to a lot of people because it was often referred to as a "preprocessor," so people got the feeling that C++ was just a clever set of macros that use the old-school C preprocessor cpp to generate C code. Nothing could have been farther from the truth. There's no way you could use the simplistic C preprocessor to convert C++ to C. Cfront was a complete, genuine compiler, with all the classic parts of a compiler: a lexer, a parser, an abstract syntax tree, etc., but instead of outputting machine language which would only work on one CPU, it output terrifying, yet perfectly portable, C code.

And that's what our ASP to PHP compiler, Thistle, does. It's a complete compiler, not a bunch of glorified regexps, which lexes and parses a VBScript/ASP program, creates an abstract syntax tree in memory, and then generates an "executable program" which happens to be in PHP. So I've been calling Thistle a "compiler" sort of to make a point, but mostly because I'm a pedantic windbag who can't resist the opportunity to teach a little lesson to the younguns who think that a compiler has to generate machine code.

Windbagginess aside, I always get a few questions about Thistle, so I'll answer the common ones here:

Q. Why didn't you use asp2php?

The quality of the "translation" produced by asp2php is nowhere near good enough to produce production quality code. On samples we fed it, almost every generated line contained an error. There are some really subtle tricks to VBScript and PHP which asp2php seems to completely punt on.

(OK, you really have to know? In VBScript you can assign an array to a variable one minute, and one minute later assign an object with a default method taking one integer argument to the same variable, so the expression a(1) would translate as $a[1] in the first case and $a->$default(1) in the second case, and it's the same expression, and this fundamental problem was completely punted on by the designers of asp2php, probably justifiably so. In a moment I'll tell you how we solved it.)

Furthermore, we had a hard-and-fast requirement that the generated PHP code had to work without further tweaks. That's because the ASP code is a living thing which changes all the time, and we constantly regenerate the PHP code automatically. If there were any manual tweaks required to get the output of Thistle to work it would dramatically increase the headache of getting a PHP build. So we implemented all kinds of conditional compilation tricks, and since we controlled the compiler, we could build in all kinds of intelligence to make sure that the generated code was exactly what we wanted.

Q. Why don't you commercialize this great Thistle thing if it's so great?

Thistle is a compiler that only has to compile one program: FogBugz. That means it doesn't need to include any logic for any features that FogBugz doesn't use. There are lots of library functions we don't have to translate because we never call them. And there are conventions in our code that we always adhere to that allow the compiler to take big shortcuts. That means Thistle would not work so well for anyone else off-the-shelf.

Remember the problem of how to translate a(1), which could mean "look up the 2nd element of array a" or "call the default method of the object a passing the argument 1" depending on what type a contains at runtime? This really matters, because we use arrays, and because we use the built-in class RecordSet all over the place, doing things like rs(1) which is short for rs.Item(1).Value, and since VBScript is latebound there is no way to know what code to generate in PHP until runtime, and that's too late! The only correct thing to do in PHP would be to generate code that checks the type of a, and decides, at runtime, whether to do an array lookup or a method call. This is messy and slow and would suck big rocks in the kinds of tight loops where you tend to be using arrays.

How did we fix it? Well, thanks to Hungarian notation, so callously dissed by developers who do would not recognize a superb coding convention if it walked up to them on the Shanghai Maglev train and shook their pants leg, every recordset at Fog Creek starts with the letters "rs". And Thistle looks for the rs and say, "ah, this is a recordset, you're not looking for an array value, you're calling the default method," and generates fast code. Based on your age you will either call this an evil hack (if you're young) or an elegant hack (if you're old); in either case it's a huge optimization made possible by the fact that Thistle only has one program to compile. Outside of Fog Creek it wouldn't work. All hail Hungarian notation!

One more reason we don't commercialize it: it's a competitive advantage to us allowing us to create server-side applications that run on Windows OR Unix (or Mac or Solaris or Linux) with a single source tree. But that's secondary.

Q. Are you serious about Hungarian notation?

Yes, it's standard at Fog Creek. We try to use something called Apps Hungarian notation, as invented by Simonyi, not the grotesque bastard Systems Hungarian notation, misinterpreted by Petzold and the entire Windows team. You can still read the original paper.

In Apps Hungarian you still have the prefixes, but they are supposed to add semantic information, not merely repeat the type of a variable. So for example when you have a buffer size, the variable should be named "cb", which means "count of bytes". Whenever you're writing C++ code that needs to deal with all kinds of strings, it's really nice to be able to look at the variable name, and if it's a psz, you know, oh, look, it's a pointer to a null-terminated string, but no memory is allocated! Or if it starts with rgch, you know it's an array of characters, with memory allocated, or rgwch, you know it's an array of Unicode (wide) chars, or if it starts with ix you know it's an index to something (or a primary key in SQL), etc. etc. It's really nice when you're manipulating unicode that cch means "count of chars" while cb means "count of bytes" since a char is not a byte, and if you ever got confused, the variable name sorts you out. And it's really nice when you don't remember if you have a pointer to a string or a pointer to a pointer to a string and the "p" or "pp" at the beginning sorts you out, too. Anyway, Hungarian is widely and correctly reviled by people who don't see the point of putting dw in front of all your longs, although even that was useful in its day, but using "c" for a count or "ix" for an index is quite helpful.

In SQL tables we use very light Hungarian to indicate the type of fields: s for strings, dt for date/time, c for a count, ix for a key, n for a number that is not a count of something or a key of some sort. The most useful aspect of this is that if a table is named Bug then you can be certain its primary key is ixBug, so whenever you're looking at SQL anywhere in the database if you see ixBug you know it's a foreign key pointing to the Bug table without bothering to look it up. And now without even thinking if you see this snippet of SQL: "join Bar on Foo.ixBar = Bar.ixBar" you know exactly what's going on, and if you see "join Bar on Foo.ixFred = Bar.ixBar" then you know it's probably wrong code because with a little bit of practice the mismatched ixFred/ixBar just jumps out at you.

I'm especially driven crazy by people who keep blabbing on their blogs that Hungarian makes code "less" readable. Yes, it's less readable if you don't know the convention, but it takes about 10 minutes to learn the convention and all the important prefixes, and people who whine about Hungarian making code "less" readable have obviously never done this. If you know what the prefixes mean Hungarian makes code more readable because every variable gives you a little bit of free, extra useful information you don't have to go look up.

Q. How did you handle all the runtime libraries ASP provides?

We reimplemented them -- either in VBScript, which gets translated to PHP, or directly in PHP. In most cases there is a very close PHP equivalent so the reimplementation is one line.

Q. What about COM objects?

That was the most work. For our own COM objects we wrote portable C++ code and we compiled that to .so's for Unix. For one of the major third-party COM objects that we use, we reimplemented it completely in PHP using a built-in PHP library. For the built-ins like Server, Response, Request, Recordset, etc. we created our own classes which usually just call underlying PHP functionality. And our implementation of Server.CreateObject knows all the tricks and just implements a giant select statement that constructs the right kind of object so we don't need an equivalent of COM class factories.

Q. Was it worth it?

Probably. FogBugz for Unix is about 10% of our sales. The effort to create it was probably 20% above the effort to create FogBugz for Windows alone. A lot of our customers are happier that we run on Unix even if they're using the Windows version, because they know they could switch servers at any time. And finally we have a certain amount of confidence that if Microsoft ever decided to stop making VBScript/ASP work on their server OS, it would be a very simple matter to change Thistle to generate the new flavor of the week ... be it Ruby.Net, Eiffel, C#, or this Language.

Tune in again tomorrow, when I'll talk about all the other stuff we did for FogBugz 4.0 to make it a complete product, not just a computer program.


The Road to FogBugz 4.0: Part IV

Thursday, March 31, 2005

A ridiculously small portion of the energy it takes to make a commercial software product actually goes into the writing of lines of code. I would estimate that out of every 100 calories expended by the Fog Creek team:

25 calories are spent on customer service
55 calories are spent on debugging, beta testing, and minor tweaks
8 calories are spent on marketing, including the Fog Creek website
5 calories are spent reading college kids' resumes and interviewing said college kids
5 calories are spent on code that never ships, such as the online demo and the online store

Leaving just:

2 calories spent on actually writing new lines of code that ship to a customer.

This, I think, helps explain why so many software companies started by programmers fail: programmers are really good at writing news line of code, they might be good at debugging, but nobody ever taught them how to do marketing or customer service, and they are probably horrible at graphic design.

Before a consumer will buy your software product, they'll evaluate all kinds of things to decide if it's a real product that will meet their needs:

  • They'll evaluate the quality of your website.
  • They'll look for discussion groups to see if people are actually using the product, and if the people who have problems are getting prompt resolutions.
  • They'll look to see if there is a third-party support infrastructure, like books.
  • They'll evaluate your reputation.
  • They'll search the web to see if you have positive buzz.
  • They'll see how long you've been around, and they'll try to evaluate whether you're profitable and successful and likely to stay around to continue supporting the product.
  • Oh, and if they have a few minutes left over, they might actually look at your product itself to see if it works.

And that explains why people buy overpriced but useless shelfware that costs $millions and doesn't really work. They rationalize that if they install it, they'll be shielded from criticism and other mercurial notes of anger.

Anyway, before we could launch FogBugz, even after the final shipping bits were perfect and ready to go, we had a list of things we wanted to have ready to make the product experience complete. For FogBugz 4.0 the big things were:

  1. Professional-quality graphic design in the user interface
  2. An online demo that included every feature of the product
  3. An online movie that introduced the product for couch potatoes
  4. A great marketing website
  5. Getting at least one book about FogBugz into bookstores
  6. Making a real physical product available on CD-ROM

Graphic Design

Human emotions can be really, really superficial. In particular people ridiculously overvalue aesthetics and beauty when evaluating products. It's one of the reasons iPods, and, for that matter, Keanu Reeves, are so successful. If you have a product that is functional but not beautiful you have a really, really, really, really, really, really, really, really, really, really big handicap to overcome against the product that is beautiful but maybe not so functional.

Given the importance of this I spent days looking at portfolios of web designers. I happened upon CSS Zen Garden, which was new at the time, the effort of Vancouver-based graphic designer Dave Shea who wanted to prove that CSS did not imply ugly boxy design. In those days there were only a few designs available on Zen Garden and a lot of them were done by Dave himself, but they were all stunning. (Here's one I particularly liked).

So we hired Dave to do a redesign of FogBugz.

Before:

After:

OK, now all the programmers in the audience are saying, "what? I don't get it?!" I know. I don't either. But the new user interface looks and feels, well, 5% more "quality," whatever the heck that means, and it's all very nebulous and vague but I can tell ya everyone has been saying it's the cat's pajamas, so I'm happy.

Online Demo

The online demo speaks for itself, but we had to do a lot of work to make it possible to host hundreds or thousands of instances of FogBugz in one directory.

In particular we were afraid that one spark of good publicity would result in hundreds or thousands of people setting up new trial accounts simultaneously, which would whack the poor servers and result in a lot of potential customers thinking FogBugz was slow simply because they were trying it on an overloaded server. So we changed the trial signup procedure to include a throttle that, under peak load, allows us to slow down the creation of new trials. Occasionally, it may mean that a potential customer has to wait a few hours to try out the software, but it's better than everybody having a bad experience.

The Online Movie

One product I just can't stop singing the praises of is Camtasia Studio, by the folks at TechSmith in Okemos, Michigan. It lets you "film" your computer screen, all in software of course, then edit the film, add narration, then squeeze it down to a very compact flash movie you can put on the web. This is a beautiful piece of software. It does exactly what you need, works the first time, and comes with great documentation you'll never need.

I used Camtasia Studio to record an introduction to FogBugz 4.0 in movie form. The whole project took me about one day. If I had to make another movie, it would only take a couple of hours, as I learned many tricks along the way. For example the first recordings I made sounded terrible. Our offices are in New York City and a lot of street noise drifts up to us here on the 18th floor. It's not enough to notice during the day but when you're trying to record something, the muted sirens in the background are shockingly unprofessional! To make things worse, we have hardwood floors throughout the office and don't have those yucky acoustic tiles on the ceiling, so the office is acoustically live in a way which made the movie sound like I was in a big abandoned warehouse, down by the river, waiting for Kojak to arrive with a lollypop.

I gathered up all the rugs and pillows I could find at Fog Creek and moved them all into my office. With a big rug on the floor and giant sofa pillows lining every wall, the sound quality was vastly improved. In fact my office was so pleasant with all the sound-absorbing stuff that I realized it's not enough to have quiet offices; you want to have acoustically dead offices as well. For our next office space I think I'll specify thick carpeting for the private offices, even if it's not as cool as wood floors. We're seriously considering hiring Aviva Stanoff to come up with some kind of framed pillow arty thing to put on the walls to deaden the sound even more.

The Website

Um, yeah. We threw this together at the last minute. We did manage to find a few good-looking FogBugz customers to give us testimonials so we could replace the crappy stock photography with pictures of actual humans, and Dmitri found a great old picture of some craftsmen making violins. I spent a lot of time putting together a screenshot tour because a lot of people look for the screenshots first when they're learning about a new product.

FogBugz, The Book

Different people learn different ways. I always want to dive in and start using a product, but some people prefer learning about a product in a classroom, and others prefer to read a book. I've even heard people say they won't use any development tool until there's a genuine book about it in bookstores. A real book from a real publishing house is one of those things that makes a product "feel" real.

I had been incredibly impressed by Mike Gunderloy's classic book Coder to Developer, about all the things that a full-fledged software developer needs to do above and beyond the basic work of writing lines of code, and Mike has been working with FogBugz since version 1.0, so I persuaded him to write a book, which Apress published.

The reason I like Mike's book so much is that it's not just a rehash of the online documentation, which is pretty detailed to begin with, and it's not just a big wasted list of steps for filling out dialog boxes. Instead Mike's book focuses on how to manage software projects using FogBugz effectively, so it adds a lot of useful material. In fact I liked the book so much I ordered a thousand copies, which we're selling along with FogBugz. Probably 20% of new FogBugz customers buy a copy or two of the book, and it's also good to have around as a handout for potential large customers.

(By the way, if you're a Mike Gunderloy fan like me, you'll be delighted to see that he has recently relaunched his website Larkware as a daily source of news about programming technologies.)

FogBugz, The CD-ROM

One of the nicest things about selling software is that there is no physical product, but I'm kind of old fashioned, and whenever I buy software online, if there's an option to receive a physical disk, I usually take it, just so I can have something physical up on the shelf that I can find if I ever need to reinstall the software. I can't tell why but somehow having the option of a physical product makes the software feel more tangible and therefore worth more money. It's a great mystery to me why this is the case but I'm sure anthopologists would not be surprised.

Since we already decided to sell physical products, i.e., the book, we needed to get set up for shipping physical products anyway, which meant inventory, postage meters, UPS accounts, and all that stuff, so I figured, against my better judgement, that it couldn't hurt to make CD-ROMs available as well.

The FogBugz CD-ROM is entirely manufactured at Fog Creek, using a robotic combination CD-ROM recorder/inkjet printer which records and prints the CD-ROM without human intervention. We put them in DVD boxes, which are a little bit classier than CD-ROM Jewel Boxes. So far it looks like we're shipping two or three a week to customers who opt to pay an extra ten bucks for physical media. It's hard to tell if it's worth it. Somehow I think that having a real packaged product makes FogBugz "seem" more real even if almost nobody orders it, but again, maybe that's just me being old fashioned.

Tomorrow, the aftermath!


The Road to FogBugz 4.0: Part V

Friday, April 01, 2005

I was really hoping to ship FogBugz for Windows and FogBugz for Unix on the same day, but it just wasn't gonna happen. There were a bunch of little details we needed to get right on the Unix side and with over a year of development we had accumulated a few little things that Thistle didn't compile perfectly. And it seemed a shame to hold up the Windows version (90% of the market) for the Unix version (10%). So on February 23, 2005, at 7:35 PM Eastern Time, FogBugz 4.0 for Windows went live.

I thought to myself, ah, now I can relax.

I have no idea why I thought that.

I should know by now that shipping software is the beginning, not the end.

We promptly got whacked. The phone lines lit up like a New Mexico license plate frame. Our weekly sales tripled. We could barely keep up with the phone calls and email inquiries. This is a good problem to have, and I don't know why I did not predict it, but even if I had, I probably wouldn't have done anything differently: I believed what Eric Sink wrote in The Hazards of Hiring: "Don't fill a position until after the need for that position is painfully clear."

And we won the Jolt Award, which is unbelievably cool (4.0 was too late so the award was for version 3.1):

In the meantime, we've got another half dozen things on our plates at Fog Creek Software. We're starting to get the kind of gigantic customers who want things just a little bit different, and they're willing to pay so much money for it you'd be insane not to give in.

We're working on rolling out a new scalable stack o' servers to handle the demands for the next five years.

We're gearing up for the coolest summer internship program you've ever heard of... our four interns will develop a brand new product from soup to nuts, and get the darn thing up and running and earning real money from real customers by the end of the summer -- something that's challenging enough without a filmmaker running around making a documentary about it, which, I hope, does not turn out like The Restaurant, Blow Out, or, god forbid, The Office. My David Brent streak is a mile long and will need to be actively surpressed. "I see myself as a friend first, probably entertainer second, and boss third..."

And we really, really need to hire more people, no matter what Eric says, which means we'll need a bigger office, so that's more work too, and life goes on.


The Wireless Web: Spacesuits Needed

Monday, July 31, 2000

I try, I really do. I held off buying a cellular phone for years. I still don't have a Palm Pilot. I managed to put off buying a stereo for long enough that my PC became a stereo. But sometimes my alternate personality takes over -- the early-adopter, gadget-loving personality, and uses my wallet to buy some unnecessary toy, play with it for two weeks, and then bury it in the closet where it belongs.

This time, alter-Joel used my money to sign me up for the latest exciting service: "Wireless Web", a.k.a. WAP service, from Verizon Wireless (mongrel spawn of BellAtlantic, GTE, and AirTouch) . This is the "service" that lets you "surf" the "web" on your cellphone. Hmmm.

I had just gotten this phone a couple of months ago, so, like most new cellphones, it already had the microbrowser capability built-in. To get access to Wireless Web, all I had to do was call Verizon and sign up. They charged me $10 a month, plus airtime. More on the cost later.

The ordering process offered a brief insight into how thoroughly mismanaged big telecom companies like Verizon can be. It was a four step process, which will be familiar to anyone who has tried to order any service from any telecom recently:

  1. Try to order the service several times. Discover that the voicemail option to order the service just hangs up on you if you call after normal business hours.
  2. Order the service during normal business hours. Promised it will work within 24 hours.
  3. 2 days later, call back. Fight with telephone menu hell.
  4. They say, "whoops!" and activate the service instantly.

Whenever you order something and they say that the service will start "within x days", you can tell that you discovered a business that has a turd-drop process in place. That's a technical term. You see, the people taking orders for the service aren't really equipped to turn it on. They just generate a list and drop it, like a turd, in a file somewhere. Once in a while someone else who really knows how to turn on the service comes along and picks up all the turds that have accumulated. Any shell script programmer will recognize this pattern: you have one program running that occasionally needs to get something done, and it does it by dropping a file in a pre-agreed place. Another program is constantly looking in that place, and when it shows up, it does its work. It's an ugly hack and no programmer would consider this to be good programming style, because so many things can go wrong.

The thing of it was, the person who took my order couldn't turn on the service, but of course, when I called to complain, they were able to turn the service on right away. So why didn't they just do that in the first place? Incompetence in business process engineering, I guess. It seems like a costly waste of money to me.

But I'm getting off the point. The service did, eventually, go on, so when I clicked "Launch Browser" on my phone, after about 5 seconds, I got the exciting main menu:

You can probably tell that typing on this thing is a bit of a pain. Here's how it works: to type a given letter, you press the key it's on once, twice, three times, or four times. "2" is A, but "22" is B. So, to type out "Vampire", you would press 8882674477733, then you'd realize that you typed "Vamphre" by mistake, then you'd play with the arrow keys to try to edit it. While you are typing, there's a button which toggles between CAPS, lower case, numbers, and punctuation modes. The "1" key can be used for space, dot, @, comma, and other punctuation, with the most common punctuation conveniently arranged first.

Apparently teenagers in Finland have gotten really good at all this. I find that even typing a one sentence email is a bit of an exercise. Better to make a voice call (and risk actually having to speak to a human!)

The screen on my Motorola Startac has room for four lines. The top line is often used for a title, and the bottom line is almost always used to describe the "softkeys": the two buttons that can do different things depending on the current mode. Reading email is not very fun with four lines, especially since one is used for the softkeys. You get about 7 words on the screen at once. Joy.

Although it's called the "Wireless Web", it's not really the web. Because of the tiny screens, regular web sites would be unusable on this service. Instead, web designers are being asked to create separate wireless web sites using a language called WML, which is a lot like HTML, the lingua-franca of web pages. There's been a bit of complaining about WML ("why isn't it just HTML?") but the truth is, the way these tiny screens work, it makes a lot of sense to have a language that is optimized for tiny screens and tiny amounts of bandwidth. Although many web sites are carping about the cost of developing two versions of their service, existing HTML pages just

You may have heard a lot of whining about the WAP protocol, which has been criticized extensively for being proprietary, stupid, and a complete attempt to reinvent TCP/IP without the benefit of, well, TCP/IP. Although the criticism is true, the truth is, the WAP protocol itself just doesn't matter to web site developers, because there's a gateway that translates the WAP requests into HTTP requests. As far as the web site developer is concerned, you just write your site using WML instead of HTML, and serve it up using a regular web server like Apache.

What Web Sites Can I Get?

This is the number one question that the cellular service providers don't want you to know the answer to. The answer is a bit tricky so pay close attention.

Some providers are providing an emasculated version of the wireless web that only allows you to get to a few "featured" sites, basically, sites that have paid for the honor of being on that service. This is the approach taken by AT&T Wireless, among others. These providers do not allow you to enter a URL for any site. It reminds me of the bad old days of AOL, when there were about 100 neat services available on AOL, but that was it. If somebody wanted to publish information for AOL, they had to make a pilgrimage to Virginia and sign a deal. But then the Internet came along, and anybody could be a publisher, and now there are a billion sites out there, and every online service had to get with the program and let people see web sites, or they would be out of business faster than you can say "Dinosaur". The unbelievable stupidity of the cellular providers that haven't learned this lesson and are trying to create their own private micro-sub-set of the wireless web is outrageous. Do not use their service unless you're the kind of person who would consider buying Internet access that only let you go to a dozen web sites.

Other cellphone companies, including Verizon, let you access any WML site. There's a way to type a URL and then make a bookmark, which is handy. I've made bookmarks for Google, Amazon, PayPal, and Visto (which I use to check my regular email). On my phone, there's only room for 10 bookmarks, and you get to them by holding down one of the number keys. For example, if I hold down the 3 key for a few seconds, it will go to Visto.

There are only a handful of WML sites live today, although silicon valley herd-mentality venture capitalists are so gaga about wireless everything, I'll betcha there will be a whole new generation of overcapitalized wireless sites soon. In the meantime, you can't really go to HTML sites over cellphones. Google is offering a service that translates any HTML site for you into WML, which, theoretically, lets you visit HTML sites, too. But the trouble is that the screen is very tiny, pages are limited to about 1500 bytes, and pictures and tables don't show up, so many sites are completely useless when you read them through Google's translation service.

Can you do anything useful?

Ah, that's more tricky. Here are some of the useful things I've found that I can do:

  • I used the online white and yellow pages once to find a nearby Barnes & Nobles. Of course, I could have just called Information.
  • I've checked my email at the beach. Really. It's kind of embarrassing ("dammit, man, you're on vacation"), but it's sort of fun to see all my Joel on Software fan mail flooding in. I'm using Visto.com to check email, because they have a decent WAP site and they can check your regular POP3 mailbox for you. (There's built-in email on the phone, but nobody sends email to that address, which is why I use Visto, which I recommend). I've only SENT email once from this thing, but it's just too much of a pain, and it's easier to call.
  • Reading the New York Times Book Review at the beach, I ordered three books from Amazon, using a pre-existing account, in a couple of minutes.

If you actually find anything else useful or entertaining you can do with this wireless web stuff, that's actually better than just making a phone call, email me, OK?

Is it worth it?

Ah, and herein lies the rub. Are you sitting down?

Yep, that's right, my phone bill was $284 for the first month of playing around with this service, and I didn't even play with it that much. I tried to analyze the bill to figure out why it was so much. I had used 147 "minutes" of data time during the month. Actually, I probably didn't use close to that much, but cellular providers love to round up to the nearest minute. And the crappy cellular system tends to knock you offline a lot. When this happens, the phone says "Connecting" for about 5 seconds and then reconnects you. But it means that you often end up paying for the same minute twice:

I had a bunch of free minutes with my calling plan, but I was over the limit, so many of these minutes were 35 cents each. After this bill, I think I'm going to limit my use to real emergencies like ordering cookbooks. I just can't afford to explore any more.

A couple of cellular providers have been advertising "free wireless web." This is pretty dishonest. What they mean is usually that they'll waive the signup charge, or even waive the monthly fee. But considering that they charge for airtime, and they round up the minutes gleefully, the airtime can be pretty damn expensive. If you sign up for wireless web, make sure that you have enough free minutes on your calling plan.

So, will wireless web take off?

In 1992, I was in the library at the University of Washington. They had set up a terminal connected to a new service called Gopher, which was, in many ways, the precursor to the web. It was very exciting. You could go to gopher sites for lots of universities all over the world. Some of them even had their course listings online! I remembered thinking about all the possibilities. When I saw Mosaic for the first time, there wasn't much you could do or see, but I realized that it was only a matter of time before this thing exploded.

Looking at wireless web today, I get that same gut feeling as I did when I looked at the web, way back when: it looks like a ghost town, without very many useful things yet. You can make all kinds of claims about its weakness (it's slow, expensive, tiny, and has no sites...), but then again, all those claims were true in the early days of the web, too, and look where we are today. Every VC on the planet has heard the story of Jim Clark discovering Mosaic a million times. Every VC and Silicon Valley entrepreneur has been searching for "The Next Internet" for so long that as soon as something comes along that sounds like it might be The Next Big Thing, they jump in.

Will history repeat itself? Maybe, maybe not. The web was truly a revolution in the way that the printing press was a revolution, but wireless web seems to be just an incremental step, not a revolution. In any case, you can debate all night about whether there will ever be truly useful wireless web services, but I'm starting to think that the high price charged by the cellular providers is going to choke off the oxygen to this fledgling service before it can get started.

Remember that when the web started, most Internet access was unmetered: you paid a low monthly fee no matter how long you used the service. But as the wireless web gets started, the fees are outrageous, and the early adopters are going to get reamed by their first phone bill. If you think you can make a profit in a wireless world in which the telcos are sucking out all the oxygen, well, you had better bring a spacesuit.

Further reading:

  • Jakob Nielsen has an excellent summary. Jakob Nielsen is the best.
  • Africa News Online - Bursting The WAP Bubble - a perspective from South Africa
  • W* Effect Considered Harmful - an engineering perspective criticizing WAP from a detailed technical point of view
  • The WAP Trap - another detailed technical criticism, claiming that the existing technology is fine and we don't need no stinkin' WAP, thank you.


Things You Should Never Do, Part I

Thursday, April 06, 2000

Netscape 6.0 is finally going into its first public beta. There never was a version 5.0. The last major release, version 4.0, was released almost three years ago. Three years is an awfully long time in the Internet world. During this time, Netscape sat by, helplessly, as their market share plummeted.

It's a bit smarmy of me to criticize them for waiting so long between releases. They didn't do it on purpose, now, did they?

Well, yes. They did. They did it by making the single worst strategic mistake that any software company can make:

They decided to rewrite the code from scratch.

Netscape wasn't the first company to make this mistake. Borland made the same mistake when they bought Arago and tried to make it into dBase for Windows, a doomed project that took so long that Microsoft Access ate their lunch, then they made it again in rewriting Quattro Pro from scratch and astonishing people with how few features it had. Microsoft almost made the same mistake, trying to rewrite Word for Windows from scratch in a doomed project called Pyramid which was shut down, thrown away, and swept under the rug. Lucky for Microsoft, they had never stopped working on the old code base, so they had something to ship, making it merely a financial disaster, not a strategic one.

We're programmers. Programmers are, in their hearts, architects, and the first thing they want to do when they get to a site is to bulldoze the place flat and build something grand. We're not excited by incremental renovation: tinkering, improving, planting flower beds.

There's a subtle reason that programmers always want to throw away the code and start over. The reason is that they think the old code is a mess. And here is the interesting observation: they are probably wrong. The reason that they think the old code is a mess is because of a cardinal, fundamental law of programming:

It's harder to read code than to write it.

This is why code reuse is so hard. This is why everybody on your team has a different function they like to use for splitting strings into arrays of strings. They write their own function because it's easier and more fun than figuring out how the old function works.

As a corollary of this axiom, you can ask almost any programmer today about the code they are working on. "It's a big hairy mess," they will tell you. "I'd like nothing better than to throw it out and start over."

Why is it a mess?

"Well," they say, "look at this function. It is two pages long! None of this stuff belongs in there! I don't know what half of these API calls are for."

Before Borland's new spreadsheet for Windows shipped, Philippe Kahn, the colorful founder of Borland, was quoted a lot in the press bragging about how Quattro Pro would be much better than Microsoft Excel, because it was written from scratch. All new source code! As if source code rusted.

The idea that new code is better than old is patently absurd. Old code has been used. It has been tested. Lots of bugs have been found, and they've been fixed. There's nothing wrong with it. It doesn't acquire bugs just by sitting around on your hard drive. Au contraire, baby! Is software supposed to be like an old Dodge Dart, that rusts just sitting in the garage? Is software like a teddy bear that's kind of gross if it's not made out of all new material?

Back to that two page function. Yes, I know, it's just a simple function to display a window, but it has grown little hairs and stuff on it and nobody knows why. Well, I'll tell you why: those are bug fixes. One of them fixes that bug that Nancy had when she tried to install the thing on a computer that didn't have Internet Explorer. Another one fixes that bug that occurs in low memory conditions. Another one fixes that bug that occurred when the file is on a floppy disk and the user yanks out the disk in the middle. That LoadLibrary call is ugly but it makes the code work on old versions of Windows 95.

Each of these bugs took weeks of real-world usage before they were found. The programmer might have spent a couple of days reproducing the bug in the lab and fixing it. If it's like a lot of bugs, the fix might be one line of code, or it might even be a couple of characters, but a lot of work and time went into those two characters.

When you throw away code and start from scratch, you are throwing away all that knowledge. All those collected bug fixes. Years of programming work.

You are throwing away your market leadership. You are giving a gift of two or three years to your competitors, and believe me, that is a long time in software years.

You are putting yourself in an extremely dangerous position where you will be shipping an old version of the code for several years, completely unable to make any strategic changes or react to new features that the market demands, because you don't have shippable code. You might as well just close for business for the duration.

You are wasting an outlandish amount of money writing code that already exists.

Is there an alternative? The consensus seems to be that the old Netscape code base was really bad. Well, it might have been bad, but, you know what? It worked pretty darn well on an awful lot of real world computer systems.

When programmers say that their code is a holy mess (as they always do), there are three kinds of things that are wrong with it.

First, there are architectural problems. The code is not factored correctly. The networking code is popping up its own dialog boxes from the middle of nowhere; this should have been handled in the UI code. These problems can be solved, one at a time, by carefully moving code, refactoring, changing interfaces. They can be done by one programmer working carefully and checking in his changes all at once, so that nobody else is disrupted. Even fairly major architectural changes can be done without throwing away the code. On the Juno project we spent several months rearchitecting at one point: just moving things around, cleaning them up, creating base classes that made sense, and creating sharp interfaces between the modules. But we did it carefully, with our existing code base, and we didn't introduce new bugs or throw away working code.

A second reason programmers think that their code is a mess is that it is inefficient. The rendering code in Netscape was rumored to be slow. But this only affects a small part of the project, which you can optimize or even rewrite. You don't have to rewrite the whole thing. When optimizing for speed, 1% of the work gets you 99% of the bang.

Third, the code may be doggone ugly. One project I worked on actually had a data type called a FuckedString. Another project had started out using the convention of starting member variables with an underscore, but later switched to the more standard "m_". So half the functions started with "_" and half with "m_", which looked ugly. Frankly, this is the kind of thing you solve in five minutes with a macro in Emacs, not by starting from scratch.

It's important to remember that when you start from scratch there is absolutely no reason to believe that you are going to do a better job than you did the first time. First of all, you probably don't even have the same programming team that worked on version one, so you don't actually have "more experience". You're just going to make most of the old mistakes again, and introduce some new problems that weren't in the original version.

The old mantra
Some feedback , including a response by a very senior engineer formerly of Netscape. Also, Seth Gordon emailed me some great tips for reading other people's source code.


Three Wrong Ideas From Computer Science

Tuesday, August 22, 2000

Not to rain on everybody's parade, but there are three important ideas from computer science which are, frankly, wrong, and people are starting to notice. Ignore them at your peril.

I'm sure there are more, but these are the three biggies that have been driving me to distraction:

  1. The difficult part about searching is finding enough results,
  2. Anti-aliased text looks better, and
  3. Network software should make resources on the network behave just like local resources.

Well, all I can say is,

  1. Wrong,
  2. Wrong,
  3. WRONG!

Let us take a quick tour.

Searching

Most of the academic work on searching is positively obsessed with problems like "what happens if you search for 'car', and the document you want says 'automobile'".

Indeed there is an awful lot of academic research into concepts like stemming, in which the word you searched for is de-conjugated, so that searching for "searching" also finds documents containing the word "searched" or "sought".

So when the big Internet search engines like Altavista first came out, they bragged about how they found zillions of results. An Altavista search for Joel on Software yields 1,033,555 pages. This is, of course, useless. The known Internet contains maybe a billion pages. By reducing the search from one billion to one million pages, Altavista has done absolutely nothing for me.

The real problem in searching is how to sort the results. In defense of the computer scientists, this is something nobody even noticed until they starting indexing gigantic corpora the size of the Internet.

But somebody noticed. Larry Page and Sergey Brin over at Google realized that ranking the pages in the right order was more important than grabbing every possible page. Their PageRank algorithm is a great way to sort the zillions of results so that the one you want is probably in the top ten. Indeed, search for Joel on Software on Google and you'll see that it comes up first. On Altavista, it's not even on the first five pages, after which I gave up looking for it.

Anti-aliased text

Antialiasing was invented way back in 1972 at the Architecture Machine Group of MIT, which was later incorporated into the famous Media Lab. The idea is that if you have a color display that is low resolution, you might as well use shades of grey to create the "illusion" of resolution. Here's how that looks:

Notice that the normal text on the left is nice and sharp, while the antialiased text on the right appears to be blurred on the edges. If you squint or step back a little bit, the normal text has weird "steps" due to the limited resolution of a computer display. But the anti-aliased text looks smoother and more pleasant.

So this is why everybody got excited about anti-aliasing. It's everywhere, now. Microsoft Windows even includes a checkbox to turn it on for all text in the system.

The problem? If you try to read a paragraph of antialiased text, it just looks blurry. There's nothing I can do about it, it's the truth. Compare these two paragraphs:

The paragraph on the left is not antialiased; the one on the right was antialiased using Corel PHOTO-PAINT. Frankly, antialiased text just looks bad.

Somebody finally noticed this: the Microsoft Typography group. They created several excellent fonts like Georgia and Verdana which are "designed for easy screen readability." Basically, instead of creating a high-resolution font and then trying to hammer it into the pixel grid, they finally accepted the pixel grid as a "given" and designed a font that fits neatly into it. Somebody didn't notice this: the Microsoft Reader group, which is using a form of antialiasing they call "ClearType" designed for color LCD screens, which, I'm sorry, still looks blurry, even on a color LCD screen.

(Before I get lots of irate responses for the graphics professionals among my readers, I should mention that anti-aliasing is still a great technique for two things: headlines and logos, where the overall appearance is more important than the sustained readability; and pictures. Antialiasing is a great way to scale photographic images to smaller sizes.)

Network Transparency

Ever since the first networks, the "holy grail" of networking computing has been to provide a programming interface in which you can access remote resources the same way as you access local resources. The network becomes "transparent".

One example of network transparency is the famous RPC (remote procedure call), a system designed so that you can call procedures (subroutines) running on another computer on the network exactly as if they were running on the local computer. An awful lot of energy went into this. Another example, built on top of RPC, is Microsoft's Distributed COM (DCOM), in which you can access objects running on another computer as if they were on the current computer.

Sounds logical, right?

Wrong.

There are three very major differences between accessing resources on another machine and accessing resources on the local machine:

  1. Availability,
  2. Latency, and
  3. Reliability.

When you access another machine, there's a good chance that machine will not be available, or the network won't be available. And the speed of the network means that it's likely that the request will take a while: you might be running over a modem at 28.8kbps. Or the other machine might crash, or the network connection might go away while you are talking to the other machine (when the cat trips over the phone cord).

Any reliable software that uses the network absolutely must take this into account. Using programming interfaces that hide all this stuff from you is a great way to make a lousy software program.

A quick example: suppose I've got some software that needs to copy a file from one computer to another. On the Windows platform, the old "transparent" way to do this is to call the usual CopyFile method, using UNC names for the files such as \\SERVER\SHARE\Filename.

If all is well with the network, this works nicely. But if the file is a megabyte long, and the network is being accessed over a modem, all kinds of things go wrong. The entire application freezes while a megabyte file is transferred. There is no way to make a progress indicator, because when CopyFile was invented, it was assumed that it would always be "fast". There is no way to resume the transfer if the phone connection is lost.

Realistically, if you want to transfer a file over a network, it's better to use an API like FtpOpenFile and its related functions. No, it's not the same as copying a file locally, and it's harder to use, but this function was built with the knowledge that network programming is different than local programming, and it provides hooks to make a progress indicator, to fail gracefully if the network is unavailable or becomes unavailable, and to operate asynchronously.

Conclusion: the next time someone tries to sell you a programming product that lets you access network resources the same was as you access local resources, run full speed in the opposite direction.


Top Five (Wrong) Reasons You Don't Have Testers

Sunday, April 30, 2000

In 1992, James Gleick was having a lot of problems with buggy software. A new version of Microsoft Word for Windows had come out, which Gleick, a science writer, considered to be awful. He wrote a lengthy article in the Sunday New York Times Magazine which could only be described as a flame, skewering the Word team for being unresponsive to the requests of customers, and delivering an enormously buggy product.

Later, as a customer of a local Internet provider Panix (which also happens to be my Internet provider), he wanted a way to automatically sort and filter his mail. The UNIX tool for doing this is called procmail, which is really arcane and has the kind of interface that even the most hardcore UNIX groupies will admit is obscure.

Anyway, Mr. Gleick inadvertently made some kind of innocent typo in procmail which deleted all his email. In a rage, he decided that he was going to create his own Internet access company. Hiring Uday Ivatury, a programmer, he created Pipeline, which was really quite a bit ahead of its time: it was the first commercial provider of Internet access with any kind of graphical interface.

Now, Pipeline had its problems, of course. The very first version didn't use any kind of error correction protocol, so it had a tendency to garble things up or crash. Like all software, it had bugs. I applied for a job at Pipeline in 1993. During the interview, I asked Mr. Gleick about the article he wrote. "Now that you're on the other side of the fence," I asked, "do you have a bit more of an appreciation for the difficultly of creating good software?"

Gleick was unrepentant. He denied that Pipeline had any bugs. He denied that it was anything as bad as Word. He told me: "one day, Joel, you too will come to hate Microsoft." I was a little bit shocked that his year of experience as a software creator, not merely a software user, hadn't given him a smidgen of appreciation for how hard it is to really get bug-free, easy to use software. So I fled, turning down the job offer. (Pipeline was bought out, by PSI, the strangest Internet provider on earth, and then unceremoniously taken out and shot.)

Software has bugs. CPUs are outrageously finicky. They absolutely refuse to deal with things that they weren't taught to deal with explicitly, and they tend to refuse in the most childish of ways. When my laptop is away from home, it tends to crash a lot because it can't find the network printer it's used to finding. What a baby. It probably comes down to a single line of code somewhere with a teensy tiny almost insignificant bug in it.

Which is why you positively, absolutely, need to have a QA department. You are going to need 1 tester for every 2 programmers (more if your software needs to work under a lot of complicated configurations or operating systems). Each programmer should work closely with a single tester, throwing them private builds as often as necessary.

The QA department should be independent and powerful, it must not report to the development team, in fact, the head of QA should have veto power over releasing any software that doesn't meet muster.

My first real software job was at Microsoft; a company that is not exactly famous for its high quality code, but which does nonetheless hire a large number of software testers. So I had sort of assumed that every software operation had testers.

Many do. But a surprising number do not have testers. In fact, a lot of software teams don't even believe in testing.

You would think that after all the Quality mania of the 80s, with all kinds of meaningless international "quality" certifications like ISO-9000 and buzzwords like "six-sigma", managers today would understand that having high quality products makes good business sense. In fact, they do. Most have managed to get this through their heads. But they still come up with lots of reasons not to have software testers, all of which are wrong.

I hope I can explain to you why these ideas are wrong. If you're in a hurry, skip the rest of this article, and go out and hire one full-time tester for every two full-time programmers on your team.

Here are the most common boo-hoo excuses I've heard for not hiring testers:

1. Bugs come from lazy programmers.

"If we hire testers", this fantasy goes, "the programmers will get sloppy and write buggy code. By avoiding testers, we can force the programmers to write correct code in the first place."

Sheesh. If you think that, you either have never written code, or you are remarkably dishonest about what writing code is like. Bugs, by definition, leak out because programmers did not see the bug in their own code. A lot of times it just takes a second set of eyes to see a bug.

When I was writing code at Juno, I tended to exercise my code the same way every time ... I used my own habits, relying on the mouse a lot. Our marvelous, vastly overqualified tester had slightly different habits: she did more things with the keyboard (and actually rigorously tested the interface using every possible combination of inputs). This quickly uncovered a whole slew of bugs. In fact at times she actually told me that the interface flatly didn't work, 100% did not work, even though it always worked for me. When I watched her repro the bug I had one of those whack-your-forehead moments. Alt! You're holding down the Alt Key! Why didn't I test that?

2. My software is on the web. I can fix bugs in a second.

Bwa ha ha ha ha! OK, it's true, web distribution lets you distribute bug fixes much faster than the old days of packaged software. But don't underestimate the cost of fixing a bug, even on a web site, after the project has already frozen. For one thing, you may introduce even more bugs when you fix the first one. But a worse problem is that if you look around at the process you have in place for rolling out new versions, you'll realize that it may be quite an expensive proposition to roll out fixes on the web. Besides the bad impression you will make, which leads to:

3. My customers will test the software for me.

Ah, the dreaded "Netscape Defense". This poor company did an almost supernatural amount of damage to its reputation through their "testing" methodology:

  1. when the programmers are about halfway done, release the software on the web without any testing.
  2. when the programmers say they are done, release the software on the web without any testing.
  3. repeat six or seven times.
  4. call one of those versions the "final version"
  5. release .01, .02, .03 versions every time an embarrassing bug is mentioned on c|net.

This company pioneered the idea of "wide betas". Literally millions of people would download these unfinished, buggy releases. In the first few years, almost everybody using Netscape was using some kind of pre-release or beta version. As a result, most people think that Netscape software is really buggy. Even if the final release was usually reasonably unbuggy, Netscape had so doggone many people using buggy versions that the average impression that most people have of the software was pretty poor.

Besides, the whole point of letting "your customers" do the testing is that they find the bugs, and you fix them. Unfortunately, neither Netscape, nor any other company on earth, has the manpower to sift through bug reports from 2,000,000 customers and decide what's really important. When I reported bugs in Netscape 2.0, the bug reporting website repeatedly crashed and simply did not let me report a bug (which, of course, would have gone into a black hole anyway). But Netscape doesn't learn. Testers of the current "preview" version, 6.0, have complained in newsgroups that the bug reporting website still just doesn't allow submissions. Years later! Same problem!

Of those zillions of bug reports, I would bet that almost all of them were about the same set of 5 or 10 really obvious bugs, anyway. Buried in that haystack will be one or two interesting, difficult-to-find bugs that somebody has gone to the trouble of submitting, but nobody is looking at all these reports anyway, so it is lost.

The worst thing about this form of testing is the remarkably bad impression you will make of your company. When Userland released the first Windows version of their flagship Frontier product, I downloaded it and started working through the tutorial. Unfortunately, Frontier crashed several times. I was literally following the instructions exactly as they were printed in the tutorial, and I just could not get more than 2 minutes into the program. I felt like nobody at Userland had even done the minimum amount of testing, making sure that the tutorial works. The low perceived quality of the product turned me off of Frontier for an awfully long time.

4. Anybody qualified to be a good tester doesn't want to work as a tester.

This one is painful. It's very hard to hire good testers.

With testers, like programmers, the best ones are an order of magnitude better than the average ones. At Juno, we had one tester, Jill McFarlane, who found three times as many bugs as all four other testers, combined. I'm not exaggerating, I actually measured this. She was more than twelve times more productive than the average tester. When she quit, I sent an email to the CEO saying "I'd rather have Jill on Mondays and Tuesdays than the rest of the QA team put together".

Unfortunately, most people who are that smart will tend to get bored with day-to-day testing, so the best testers tend to last for about 3 or 4 months and then move on.

The only thing to do about this problem is to recognize that it exists, and deal with it. Here are some suggestions:

  • Use testing as a career move up from technical support. Tedious as testing may be, it sure beats dealing with irate users on the phone, and this may be a way to eliminate some of the churn from the technical support side.
  • Allow testers to develop their careers by taking programming classes, and encourage the smarter ones to develop automated test suites using programming tools and scripting languages. This is a heck of a lot more interesting than testing the same dialog again and again and again.
  • Recognize that you will have a lot of turnover among your top testers. Hire aggressively to keep a steady inflow of people. Don't stop hiring just because you temporarily have a full manifest, 'cause da golden age ain't gonna last.
  • Look for "nontraditional" workers: smart teenagers, college kids, and retirees, working part time. You could create a stunningly good testing department with two or three top notch full timers and an army of kids from Bronx Science (a top-ranked high school in New York) working summers in exchange for college money.
  • Hire temps. If you hire about 10 temps to come in and bang on your software for a few days, you'll find a tremendous number of bugs. Two or three of those temps are likely to have good testing skills, in which case it's worth buying out their contracts to get them full time. Recognize in advance that some of the temps are likely to be worthless as testers; send them home and move on. That's what temp agencies are for.

Here's one way not to deal with it:

  • Don't even think of trying to tell college CS graduates that they can come work for you, but "everyone has to do a stint in QA for a while before moving on to code". I've seen a lot of this. Programmers do not make good testers, and you'll lose a good programmer, who is a lot harder to replace.

And finally. The number one stupid reason people don't hire testers:

5. I can't afford testers!

This is the stupidest, and it's the easiest to debunk.

No matter how hard it is to find testers, they are still cheaper than programmers. A lot cheaper. And if you don't hire testers, you're going to have programmers doing testing. And if you think it's bad when you have testers churning out, just wait till you see how expensive it is to replace that star programmer, at $100,000 a year, who got sick of being told to "spend a few weeks on testing before we release" and moved on to a more professional company. You could hire three testers for a year just to cover the recruiter's fee on the replacement programmer.

Skimping on testers is such an outrageous false economy that I'm simply blown away that more people don't recognize it.


Top Twelve Tips for Running a Beta Test

Tuesday, March 02, 2004

Here are a few tips for running a beta test of a software product intended for large audiences -- what I call "shrinkwrap". These apply for commercial or open source projects; I don't care whether you get paid in cash, eyeballs, or peer recognition, but I'm focused on products for lots of users, not internal IT projects.

  1. Open betas don't work. You either get too many testers (think Netscape) in which case you can't get good data from the testers, or too few reports from the existing testers.
  2. The best way to get a beta tester to send you feedback is to appeal to their psychological need to be consistent. You need to get them to say that they will send you feedback, or, even better, apply to be in the beta testing program. Once they have taken some positive action such as filling out an application and checking the box that says "I agree to send feedback and bug reports promptly," many more people will do so in order to be consistent.
  3. Don't think you can get through a full beta cycle in less than eight to ten weeks. I've tried; lord help me, it just can't be done.
  4. Don't expect to release new builds to beta testers more than once every two weeks. I've tried; lord help me, it just can't be done.
  5. Don't plan a beta with fewer than four releases. I haven't tried that because it was so obviously not going to work!
  6. If you add a feature, even a small one, during the beta process, the clock goes back to the beginning of the eight weeks and you need another 3-4 releases. One of the biggest mistakes I ever made was adding some whitespace-preserving code to CityDesk 2.0 towards the end of the beta cycle which had some, shall we say, unexpected side effects that a longer beta would have fleshed out.
  7. Even if you have an application process, only about one in five people will send you feedback anyway.
  8. We have a policy of giving a free copy of the software to anyone who sends any feedback, positive, negative, whatever. But people who don't send us anything don't get a free copy at the end of the beta.
  9. The minimum number of serious testers you need (i.e., people who send you three page summaries of their experience) is probably about 100. If you're a one-person shop, that's all the feedback you can handle. If you have a team of testers or beta managers, try to get 100 serious testers for every employee that is available to handle feedback.
  10. Even if you have an application process, only one out of five testers is really going to try the product and send you feedback. So, for example, if you have a QA department with 3 testers, you should approve 1500 beta applications to get 300 serious testers. Fewer than this and you won't hear everything. More than this and you'll be deluged with repeated feedback.
  11. Most beta testers will try out the program when they first get it, and then lose interest. They are not going to be interested in retesting it every time you drop them another build unless they really start using the program every day, which is unlikely for most people. Therefore, stagger the releases. Split your beta population into four groups and each new release, add another group that gets the software, so there are new beta testers for each milestone.
  12. Don't confuse a technical beta with a marketing beta. I've been talking about technical betas, here, in which the goal is to find bugs and get last-minute feedback. Marketing betas are prerelease versions of the software given to the press, to big customers, and to the guy who is going to write the Dummies book that has to appear on the same day as the product. With marketing betas you don't expect to get feedback (although the people who write the books are likely to give you copious feedback no matter what you do, and if you ignore it, it will be cut and pasted into their book).


Two Stories

Sunday, March 19, 2000

I want to tell you two stories from my career which I think are classic illustrations of the difference between tech companies that are well-managed and tech companies that are disasters. It comes down to the difference between trusting employees and letting them get things done, versus treating them like burger flippers that need to be monitored and controlled every minute, lest they wander off and sabotage everything.

My first assignment at my first job was working at Microsoft, where I was told to come up with a new macro language strategy for Excel. Pretty soon, I had the first draft of the "Excel Basic" spec (which later evolved into Visual Basic for Applications, but that's another story). Somehow, this mysterious group of people at Microsoft called the "Application Architecture" group got wind of my spec, which must have concerned them, because for some reason they thought that they were in charge of things like macro language strategies, and they asked to see my spec.

I asked around. Who's the Application Architecture group? Nobody seemed to think they were very serious. It turns out that they were a group of just four people, recent hires with PhDs (very unusual for Microsoft). I sent them a copy of my spec and went to meet them, in case they had something interesting to say.

"Blah blah!" said one of them. "Blah blah blah, blah blah blah!" said another. I don't think they quite had anything interesting to say. They were very enamored of the idea of subclassing and sort of thought that people making macros in Excel wanted to subclass a lot of things. In any case, one of the fellows said, "Well, this is all very interesting. What's next? Who has to approve your spec?"

I laughed. Even though I had only been at Microsoft for a few months, I knew that there was no such thing as somebody approving my spec. Hell, nobody had time to read my spec, let alone approve it. The programmers were bugging me every day to get them more pages so that they could write more code. My boss (and his boss) made it very clear to me that nobody else understood macros or had time to work on macros, so whatever I did, it better be right. And here this PhD working in a strange research group at Microsoft assumed that things were a bit more formal than that.

I pretty rapidly realized that the App Architecture group knew even less than I did about macros. At least, I had talked to a handful of macro developers and some Excel old-timers to get a grip on what people actually did with Excel macros: things like recalculating a spreadsheet every day, or rearranging some data according to a certain pattern. But the App Architecture group had merely thought about macros as an academic exercise, and they couldn't actually come up with any examples of the kind of macros people would want to write. Pressured, one of them came up with the idea that since Excel already had underlining and double-underlining, perhaps someone would want to write a macro to triple underline. Yep. REAL common. So I proceeded to ignore them as diplomatically as possible.

This seemed to piss off a guy named Greg Whitten who headed up the App Architecture group. Now, Greg was something like Microsoft employee number 6. He had been around forever; nobody could quite point to anything he had done but apparently he had lunch with Bill Gates a lot and GW-BASIC was named after him. Greg called a BIG MEETING and proceeded to complain about how the Excel team (meaning me) was screwing up the macro strategy. We pressured him to come up with some specific reasons but his arguments just weren't convincing. I thought it was nice that here I was, a new hire pipsqueak right out of college, arguing with employee number 6 and apparently winning the argument. (Can you imagine that happening at a Grey Flannel Suit company?) My programming team, headed by Ben Waldman (now a VP at Microsoft) backed me up completely, which was all that really mattered, because the programming team wrote the code and thus had the final say on how things got done.

I would have been perfectly happy to leave it at that. If the Apps Architecture team needed care and feeding and wanted to argue about stuff, that was OK, I would argue with them as much as they wanted as long as they left the programmers alone to do their work. But then something even more interesting happened that blew my mind. I was sitting at lunch with some coworkers, in the Redmond sun, when Pete Higgins came up to me. At that time Pete was the general manager for Office -- I knew who he was, of course, but didn't expect that he knew me very well.

"How's it going, Joel?" he asked. "I hear you've been having some issues with the App Architecture group."

"Oh no!" I said. "Nothing I can't handle."

"Say no more," he said, "I understand." He left. By the next day the rumor had gotten back to me: the App Architecture group was disbanded. Not only that, but each member of the group was sent to a different department at Microsoft, as far apart as possible. I never heard from them again.

I was blown away, of course. At Microsoft, if you're the Program Manager working on the Excel macro strategy, even if you've been at the company for less than six months, it doesn't matter - you are the GOD of the Excel macro strategy, and nobody, not even employee number 6, is allowed to get in your way. Period.

This sends a really strong message. For one, it makes everyone that much more conscientious about their jobs. They can't hide behind the idea that "management approved their spec," since management really didn't look too closely at their spec. All management did was hire smart people and gave them something to do. For another, it makes for an extremely nice place to work. Who doesn't want to be king of their own domain? Software, by its nature, is very easy to divide into smaller and smaller components, so it's always possible to divide up responsibility among people and let people own an area. This is probably THE reason why software people love working at Microsoft.

Years passed. I found myself working at Juno, an online service and free email provider. This time, the experience was the exact opposite of my work at Microsoft. I had two programmers reporting to me, but my own manager constantly undermined my (limited) authority by going directly to my reports and giving them things to do, often without even telling me. Even for trivial requests like days off, my manager thought that it was his job to approve or disapprove the request.

After a couple of years at Juno I was working on the new user signup feature. For Juno 3.x, a major release, I was going to be in charge of a complete overhaul of the signup process. By this time, I was a relatively senior member of the technical team; I got great performance reviews, and my managers seemed to appreciate the work I was doing. But they just couldn't bring themselves to trust me. Command and control.

One part of the signup process asked the users to type in their birthday. This was just one small bit of a lengthy signup process that went on for something like 30 screens as Juno grilled you about your income, your favorite sports, how many children you have and how old they were, and about 100 other things. To make the signup process a little bit easier, I wanted to change the birthday field to be free format, so you could type "8/12/74" or "August 12, 1974" or "12 Aug 74" or whatever. (Have you used Outlook? It would work like Outlook, where you could type dates in just about any format and it would accept them).

Without going into too much detail, my manager decided he didn't like this. It became an issue of ego for him. First he yelled at the designer who was working on that page (without even telling me). Then he yelled at me. Then he reminded me every single day that I had to change it to the way he wanted it. Then he got the CEO of the company to review it, and made a big show out of getting the CEO of the company to criticize my new design. Even the CEO at Juno is perfectly happy to interfere in work done at the lowest level in the company, in fact, it's standard operating procedure.

I was furious, needless to say. It was a small thing, a matter of taste, really. Some people would prefer my way. Some people would prefer his. In either case, the message was clear: you WILL do as you are told here, dammit. It was a very command-and-conquer mentality that was more of a battle of cojones than a discussion of user interface design.

I won't say that this is the reason I left Juno, but it does illustrate the reason I left Juno: it was the idea that no matter how hard you work, no matter how smart you are, no matter whether you are 'in charge' of something or not, you have no authority whatsoever for even the tiniest thing. None. Take your damn ideas, training, brains, and intelligence, all the things we're paying you for, and shove it. And at Juno, there were plenty of managers, something like 1/4 of all the employees, and so they had plenty of times to stick their fingers into every single decision and make sure that they were in control. The contrast with Microsoft, where VP's descended from Building 9 to make it clear that you have the authority to get things done, was stark.

To some extent, Juno's hopelessly inept management process is a factor of being a New York City company, not a West Coast company, so modern styles of management haven't quite permeated. It's also a problem caused by the deep inexperience of Juno's managers, and it originates at the top - the CEO, a 29 year old who has never worked outside D. E. Shaw, who interferes in everything he can get his fingers into, including the wording on error messages that come up when things go wrong; the CTO regularly screams at his reports if they dare to question his wisdom; they take it out on the programmers, who go home and kick their dogs. Compare this to Microsoft, where things are done at the lowest level, and most managers act like their most important job is to run around the room, moving the furniture out of the way, so people can concentrate on their work.


Up the tata without a tutu

Saturday, December 02, 2000

Until yesterday, the FogBUGZ license said that you couldn't reverse engineer the program, attempt to look at the source code, or modify it in any way. Various honest people have asked how much we charge for a source license, so that they could customize a few things.

Hmmm. Why does the license say you can't change the source code? I couldn't think of a single reason. In fact I thought of a lot of counter-reasons, and immediately changed the license agreement. So now you're going to have to sit through one of those old-fuddy-duddy stories from my past.

Way back in 1995, I was working at Viacom, where a small group of us hardy pioneers were building web sites for various Viacom properties.

In those days, there were no application servers. Sybase was so clueless that if you wanted to use their database on the Internet they told you that you needed to buy a $150 client license for every user that connects to your web site. Netscape's web server was up to version 1.0.

A brave company called Illustra started telling people that their database management system was perfect for the web. You see, Illustra was designed to make it easy to add new data types by writing some C code and linking it in with their DBMS. (Any programmer who's used a DBMS will tell you that this is already sounding a bit too dangerous. C code? linked in? Oy.) This was originally intended for exciting data types like Latitude/Longitude, time series, and so on. But then the web happened. Illustra wrote something they called a "Web Blade" and linked it in. The Web Blade was a sort of half-baked system that allegedly made it possible to extract data from the database and create dynamic web pages on the fly, which was the biggest problem everybody had in 1995.

A colleague of mine at Viacom was put in charge of building an ecommerce site so that Blockbuster could sell, I kid you not, CDs on the web. (Because that's what people think of when they think of Blockbuster, right?) Anyway, he thought that Illustra would be perfect for the job. Now, Illustra cost something like $125,000, and shaking that much money loose from the Viacom Tree is like herding cats, so it took a while. My colleague taped a paper cup to his cube labeled "Illustra Fund" and collected a few dollars that way. The CTO negotiated hard and long hours with Illustra and eventually a deal was struck. We installed Illustra and got to work.

Unfortunately, disaster struck. Illustra's Web Blade was barely half baked and completely not up to the task. It crashed every few minutes. When it did run, it proved to have the only programming language I've ever seen that wasn't Turing-equivalent, if you can imagine that. The license manager kept deciding to shut you off and your site would die. Building a site with it was terrible, my colleague's annus horribilis. So when they came to me and said, "Joel, you're making a site for MTV," I said, "uh oh."

"Please can I not use Illustra?" I begged.

"Well, OK, but what are you going to use instead?" There really weren't any other app servers in those days. There was no PHP, no AOLServer with TCL stuff, perl had to fork, we didn't have penicillin, life was terrible.

And my reputation was on the line. And I decided that the scariest thing about Illustra was that when it crashed, you couldn't do anything about it. At least, if you had the source code, I thought, if Illustra crashes, well, it falls into your lap in the debugger and you can try to fix the bug. You may have to stay up all night for a week debugging someone else's code, but at least you have a chance. Whereas, without the source code, you are up the proverbial tata without a tutu.

And that's where I learned a key lesson in software architecture: for your most important, mission critical stuff, you have to use a tool that is one level lower in abstraction than ideal. For example, if you're writing a cool 3D shoot-em-up game (like Quake, around the same time period) and your key number 1 differentiator is to have the coolest 3D graphics, you do not use whatever 3D library you can find. You write your own, because it's fundamental to what you do. The people who use 3D libraries like DirectX are using them because they are trying to differentiate their games on something other than 3D performance. (Maybe the story line.)

That's when I decided not to trust anyone else's poxy application server, and decided to just write my own, in C++, using Netscape Server's low level API. Because I knew that at least, if anything went wrong, it was in my code and I could eventually fix it.

And this is one of the greatest virtues of open source / free software, even if you could afford Illustra's $125,000 piece of tata: at least if anything goes wrong, you are going to be able to fix it, somehow, and you won't get fired, and the nice-if-hyperactive people at MTV won't be all pissed off at you.

When I sit down to architect a system, I have to decide which tools to use. And a good architect only uses tools that can either be trusted, or that can be fixed. "Trusted" doesn't mean that they were made by some big company that you're supposed to trust like IBM, it means that you know in your heart that it's going to work right. I think today most Windows programmers trust Visual C++, for example. They may not trust MFC, but MFC comes with source, and so even though it can't be trusted, it can be fixed when you discover how truly atrocious the async socket library is. So it's OK to bet your career on MFC, too.

You can bet your career on the Oracle DBMS, because it just works and everybody knows it. And you can bet your career on Berkeley DB, because if it screws up, you go into the source code and fix it. But you probably don't want to bet your career on a non-open-source, not-well-known tool. You can use that for experiments, but it's not a bet-your-career kind of tool.

So I got to thinking about how to make FogBUGZ a safe bet for smart engineers. Almost by accident, it ships in source code form -- because that's how ASP pages work these days. Which doesn't bother me. There are no magical, trade-secret algorithms in bug tracking software. This stuff is not rocket science. (In fact, there are very few magical, trade-secret algorithms in any software. The fact that it's fairly easy to disassemble an executable and figure out how it works just doesn't matter as much as intellectual property lawyers think it should.) It doesn't matter to me that people look at the code or modify the code for their own use.

There's another risk when you modify source code that you bought from a vendor: when the vendor upgrades the code, you are going to have a heck of a time migrating your changes to their new version. There's something I can do to ameliorate that, too: if you find a bug in FogBUGZ and fix it, and send me the fix, I'll incorporate it into the next version. This is intended to make people feel a little bit more comfortable that (a) FogBUGZ works, and (b) if it doesn't work, in some mission-critical way, they can fix it rather than get fired, and (c) if they do have to fix it, and their fix makes sense, it will get back into the source tree so that the next version will incorporate their fixes and life will be less brutish.

By now I can hear the open-source and free software advocates practically screaming, "you silly goose! just make it open source and be done with it! open source doesn't have any of these problems!" And that's nice. But my wee company with three programmers costs $40,000 a month to operate. So we just charge for our software, and we don't apologize, because it's worth the money. We don't claim to be open source but we can make sure that FogBUGZ is a safe decision to make, by adopting two or three nice features from the open source world.


Usability Testing with Morae

Saturday, July 30, 2005

The last time I participated in formal usability testing was at a fancy lab in Colorado, custom built for the purpose at a cost of around $100,000. It was basically a television studio, complete with one-way glass, lots of special video gear, and a giant video console that would have been adequate to broadcast the Superbowl. To do usability testing for Juno, a group of us flew out to Colorado, rented cars, stayed in a hotel, ate at expensive restaurants, and generally consumed massive amounts of money so we could watch people try to sign up for our online service, and, generally, succeed.

At the other extreme, I've long been an advocate of hallway usability tests and paper prototypes, which often find some of the biggest usability problems long before they occur for about fifty cents.

Now there's a middle ground. My friends over at TechSmith in Okemos, Michigan recently released a software product called Morae which lets you use cheap webcams to set up a complete usability lab in your office without fancy equipment or one-way glass. I asked them if they would be willing to usability test their own product by running a usability test in the Fog Creek office for our new remote assistance service, and they graciously agreed.

Here's how Morae works. You set up your usability testing subject in front of a computer with a webcam and a microphone:

Then any number of people can watch the subject from their own computers:

Here Tyler is watching two screens: one is showing the helper and the other is showing the person being helped. He can see their screens, hear everything they say, and see video of the subject in the corner. We happen to have windows between the offices at Fog Creek so he can actually see the helper through his window. Let's see if I can zoom in on Tyler's screens:

In my book on UI design, I wrote about a common problem with usability tests:

In most usability tests, you prepare a list of instructions for the user. For example, if you were usability testing an Internet access provider, you might have an instruction to ?euro;œsign up for the service.?euro;? (I actually did this very usability test, several times, in my career.)

So far so good. The first user comes in, sits down, starts signing up for the service, and gets to the screen asking them how they want to pay. The user looks at me helplessly. ?euro;œDo I gotta pay for this myself??euro;?

?euro;œOh wait,?euro;? you interrupt. ?euro;œHere, use this fake credit card number.?euro;?

The sign up procedure then asks them if they would like to use a regular modem, a cable modem, or a DSL line.

?euro;œWhat do I put here??euro;? asks the user. Possibly because they don?euro;™t know the answer, but possibly because they know the answer for their computer, but they?euro;™re not using their computer, they?euro;™re using your computer, which they?euro;™ve never seen before, in a usability lab, where they?euro;™ve never been before.

To work around this problem, usability testers have started trying to do field testing. Instead of giving the subject tasks to do in a highly contrived environment, you conspire to watch the subject doing their own work at their own desk while you hide in a nearby shrub and spy on them. Morae, by the way, would be perfect for that. This method is most useful when you already have a version

The usability test worked great. Our usability test was a little bit uncommon in that we had two subjects, since Fog Creek Copilot involves a helper and a helpee, and Morae only let us hear one subject at a time from one computer. To work around this we just set up two computers with the Morae Remote Viewer so we could get the sound from both subjects.

So far this morning we've run two usability test sessions, with great results: we've already realized that 2 out of 2 helpers were confused about how to get reconnected, since the Fog Creek Copilot helper application deletes itself when you're done with it. This is a classic example of the user model not conforming to the program model ... most programs don't delete themselves! ... which is the source of virtually every usability problem. From the very first chapter of UI for Programmers:

The cardinal axiom of all user interface design:


Wall Street Survival 101

Friday, May 20, 2005

Yesterday, Dan Bricklin was interviewing me for his new podcast, and after a lot of questions about open source, which I'm afraid I answered in a way which will likely cause great consternation to people who think open source benefits programmers economically, but anyway, after all the open source questions, he asked me if I had anything to add.

"Yes!" I said. "Never, ever, ever buy bonds at retail from a full-service broker. Especially municipal bonds."

It wasn't really related to open source but it's pretty important that if you have a retail brokerage account and your broker tries to sell you bonds, you don't buy them. The typical commissions they take will tend to lop off a year of interest, so the day you buy the bond, you've lost money. But there's something even scarier with the retail bond market -- basically, a scam -- that Wall Street uses to make certain bonds look like really good investments to you, the average investor, when they're really bad investments.

Here's the puzzle I published on my website yesterday:

You have $100,000 to invest. Which government bond should you invest in?

Bond A, B or C?

Bond A pays 4.15%. If you buy this bond, you'll get a check for $4150 in interest at the end of every year for 10 years. With your last check, you'll get your $100,000 back.

Bond B pays 4.5%. You'll get a check for $4500 at the end of every year for 30 years, then with your last check, you'll get your $100,000 back.

Bond C pays even more! It's 4.75%, w00t. You will get $4750 at the end of every year for 10 years, then get your original investment back, unless the government decides they want to keep your money for a little longer, in which case you'll get another 20 years of $4750 before you get your money back.

Bond C is a little more complicated so let me explain. On the tenth anniversary, the government gets to decide whether to pay you your $100K back and owe you nothing, or keep your $100K for another twenty years and keep paying you the interest. The point being, the choice is up to the government.

But either way, with Bond C, you're getting MORE interest every year, whether they decide to keep your money for 10 years or 30.

So, it's stupid to buy bonds A or B, right? Bond C is obviously the better choice.

Right?

Hints: assume that you're a left-handed avocado farmer and therefore don't pay any tax. So ignore the tax implications. Then, assume that all the bonds are equally sound, backed by the "full faith and credit of the United States" which, in the financial world, means there is absolutely no chance of default. This is not a trick question.

Before I get started, let's clear off some distractions from the table.

  1. It has nothing to do with compounding. The bonds I described do not compound. They just pay you interest once a year. What you do with that interest is your problem. You might buy some different bonds, but who knows what interest rate you're going to get? Interest rates change all the time and are unpredictable.
  2. This is not a question about whether you're going to need the money in 10 years, or whether you're going to die in 20 years, or whether the government deficits are going to grow or shrink. I'm just asking, all else being equal, which bond is a better deal?

Now on to the answer.

Bond C looks, to almost everybody, to be a better deal. It's for a higher interest rate, so your broker can tell you little lies like, "You're getting 4.75% on that thing... tax free! What do you get at your bank? Maybe 2%?" Most people just look at the interest rate and try to invest in the thing with the highest interest rate. So for that reason alone, most people would just choose Bond C.

But a few brave intelligent investors actually try to work out the problem in their head and they reason as follows:

  • If I take bond C, there are two possibilities.
    1. The government pays me back in 10 years, or
    2. The government pays me back in 30 years.
  • In case #1 I make more money than I would have made with bond A... for ten years I'm making $4750 a year instead of $4150, and that's more!
  • In case #2 I make more money than I would have made with bond B... for thirty years I'm making $4750 a year instead of $4500, and that's more money too!
  • So in either case, I make more money!
  • Mo money, mo money, mo money, give me the more money! What could be wrong with taking a risk where in EITHER situation I'm better off?

And these people buy bond C, too.

And they are getting ripped off.

I'll try to explain why.

The secret lies on that day, 10 years in the future, where the government decides whether to pay you back right now or keep your money for an extra 20 years.

How are they going to decide?

Some people think, "Well, it depends on whether we have a bad president who keeps running deficits, so they need more money, or if we have a good president who is paying off the national debt, and they don't need the money."

No. That's not what it depends on, actually. What it depends on is only one thing: what are the interest rates on that day in the future.

Here's why.

Lets say interest rates go down. So in ten years, the interest rate goes down below 4.75%.

In that case, the government is going to pay you back on the spot, because they can borrow the same money somewhere else for cheaper. For example, if the interest rate in ten years is only 4%, the government can save $750 a year by borrowing elsewhere and paying off your loan, so they'd be crazy not to do it.

But what if interest rates go up?

In that case, the government is not going to pay you back, even if they have extra money burning a hole in their government pockets! If they don't need the money, they can just loan it to someone else. For example, if the interest rates in ten years are 6%, then the government can loan your $100,000 to someone else at 6%, making $6000 a year, and paying you your $4750 a year, and they're making a $1250 profit which they can use to give tax refunds to rich Texasshire oilmen and bomb third world countries, or whatever they want to do.

But the important point is:

  • If rates go down they will prepay.
  • If rates go up they will not prepay.

Notice it has nothing to do with whether the government "needs" your money or not in ten years. The only thing it depends on is interest rates.

Now, let's reexamine Bond C from your perspective.

You buy the bond.

Rates go up.

Now you're screwed, because you're stuck with your crappy low-interest rate bond even though everybody else is getting high interest rates.

Or let's say rates go down.

Now you're screwed, because the government pays you back, and you have all this money, and you can only invest it at the new, sucky, lower rate.

You're screwed either way. The only way you're not screwed is if rates stay approximately the same.

When you buy Bond C, you're giving up something valuable. You're giving the government an option to call your bond.

This option has monetary value, because you're agreeing to take a hit if interest rates change. And people on Wall Street can actually calculate this value.

In fact, Bond C, as of today, "at today's rates and implied volatility," was worth about $2000 less than Bonds A or B, which were of equal value "at today's rates and implied volatility."

"Implied volatility" is a fancy word to mean "the bond market's collective opinion, as reflected in the prices they are willing to pay for various things, as to how likely interest rates are to move up or down in the future.

Let me translate. Bond C, which looks like a better deal to almost everyone who doesn't know how to trade bonds, is worth $2000 less.

If you took all three of these bonds and tried to sell them today on the bond market, you'd get $100,000 for Bond A and Bond B, while you would only get $98,000 for Bond C.

How did I calculate that?

Well, I didn't. I have no idea how to calculate it. Jared calculated it for me (he works on Wall Street) by going into his office, closing the door, and banging away for the computer for a while. He was doing weird and complicated things with his Bloomberg terminal ($1500/month), spreadsheets (free with Microsoft Office), and the latest Green Day album ($8/month paid to Rhapsody). He needed a yield curve, he needed to know the implied volatility, and all kinds of math things. Smoke came out from under the door to his office. After a while he came out, rumpled of hair and shirt, and announced that Bond C was worth $2000 less.

So the retail brokers selling these bonds have snuck a sneaky derivative into the bond, and you can't calculate what it's worth unless you do this for a living, but I assure you that they have calculated it to the nearest penny and are helping themselves to a couple of percent of your money there, and the long and the short of it is that you're getting ripped off in a way you can't possibly be sophisticated enough to calculate.

This is the scam.

Most bonds sold to investors at retail by full service brokers are disguised to look like they pay a lot of interest when the truth is that the high interest rates never really compensate you for the option you're implicitly selling.

This is not a coincidence. It's by design. All else being equal, the people who sell these bonds love to sneak in prepayment options because it enables them to make it look like you're getting a higher rate of return. These bonds are designed to rip off retail investors.

But then again, full service brokers, in this day and age of low cost mutual funds and discount brokers, are really nothing more than machines for ripping off retail investors, so don't be surprised!


Wasting Money on Cats

Tuesday, September 12, 2000

Did you get this little box in the mail this week?

I did. I think it's because I'm a Wired subscriber. Inside the box was a free "cat", which is supposed to remind you of a mouse, and Jared says looks like a fox:

The little cat can be used to scan barcodes which appear in magazines like, um, Wired, for example.

And when you scan the cool barcode, presumably, your web browser goes to the Altoids web page.

Well.

The number of dumb things going on here exceeds my limited ability to grok all at once. I'm a bit overwhelmed with what a feeble business idea this is.

The cat is called Cue Cat, which I'm supposed to punctuate as so: :CueCat. That's right, the colon is a part of the name. The software that comes with this cutie says that it's made by DigitalConvergence.:Com. No, don't type the colon before the com, that's merely there to be cute.

The colonated company DigitalConvergence.:Com has, and I kid you not, 200 employees, according to their web page. That means that even on the most moderate estimate, they are burning about a million dollars a month just on salaries. Now, the auditors say that the paid circulation of Wired is around half a million, so mailing this dang thing out is going to cost about a million dollars in postage, if they're really sending it to everyone. Not to mention the cost of manufacturing the CD with the software; the cable that comes with the thing, and the :Cat itself: this is a company that is burning money at Iridium rates.

What kind of investors are willing to burn money at that rate? Don't they realize that it's a dumb idea?

I guess not. But it is. Here's why.

1. This thing is not solving a problem.

I've racked my brain, and I've tried to figure out why I would want their cat chasin' my mouse around the desk. I came up with two "problems" that it solves:

  1. typing URLs is hard. As if. Going to the Altoids web site is not a hard problem that I need solved. We're talking 7 characters to type, here.
  2. magazines can't prove to advertisers that people actually look at their ads and go to the URLs mentioned in the ad.

I think that #2 is really where the focus is, because that seems to be where the money is to be made (and that's who's going to pay for the cats, which users get for free). But #2 is a falsehood, too. People don't have cats and they won't use them. Just to direct mail a bunch of cats, free, to the subscribers of one magazine is going to cost millions, and if it's like any other direct mail, 99% of them are going straight in the trash. If 1% of Wired's subscribers install the thing, we're talking about an installed base of around 5000 cats. So if Wired thinks they are going to impress their advertisers by showing them how 13 people scanned the page and went to the web site, they're in for a rude awakening.

If you invent something that doesn't solve a problem, it better be entertaining. Is the :Cat entertaining? Let me entertain you with a quick quote from the instructions that came with this thing:

"TO CHANGE YOUR COMPUTER'S BIOS SETTINGS..."

That's where I stopped reading. Changing the BIOS settings is not my idea of entertainment. I don't think I want to install this. There is no possible benefit to the consumer, so no consumer in their right mind will use this.

2. The Cat Suffers from Chicken and Egg Syndrome

No advertiser will bother putting the Cat barcode in their ads unless a lot of people have cats installed on their computers, because, well, it's just a lot of bother and looks dumb.

Nobody will install a Cat on their computer unless they see Cat barcodes all over the place.

Conclusion: this thing will die an unhappy death of Chicken and Egg Syndrome. You can read all about this syndrome in my earlier article here. This might be a decent business if everybody had cats installed, and it would be a great business if every ad, everywhere, had barcodes, but, uh, they don't, and they don't, so it's not such a great business.

To get around the chicken/egg problem, DigitalConvergence.:Com is spending a fortune giving away the devices. If they send them to all of Wired's subscribers, well, that's half a million people. Out of 300 million people on the net, I'm not impressed. And their system lets you scan in UPC symbols, for some bizarre reason, so if you want to go to the Campbell's Tomato Soup web page, you can do that by scanning the actual can itself. Both of these are half-hearted attempts to solve the mother of all chicken and egg problems.

3. And anyway, what the heck happened to last month's dumb Wired idea?

About two months ago, Wired magazine had a different technology for going to a URL automatically from an ad. It was some kind of weird thing where you held up the page to your digital camera, took a digital picture, and ran this wacked out software that navigated your browser to the Altoids home page. So now instead of typing 7 letters I have to find my digital camera, turn it on, wait for it to boot up, take a picture of the page, turn off the camera, wait for it to flush its memory to flash, remove the flash card from the camera, take the network card out of the PCMCIA slot, put the compact flash into it's holder, plug it into the PCMCIA slot, find the picture, run the software which I previously installed, oh, don't get me started. It would be a half-hour trauma just to go to the damn Altoids web site, where you can't even buy an Altoids, for heaven's sake. Curious.

Of course, that idea died so quickly that here it is, two months later, and there's no sign of it in the pages of this month's Wired. A mere flash in the primordial soup. I can't even remember what the damn thing was called. (My readers inform me that it's called the Digimarc MediaBridge.)

Bottom line: if you're working for a company that's spending millions of dollars trying to get people to do something with absolutely no benefit to them, and which suffers from chicken and egg problems, don't be counting your stock options just yet.

[June 16, 2001: Digital Convergence lays off most employees.]


Whaddaya Mean, You Can't Find Programmers?

Thursday, June 15, 2000

Ask any software CEO these days what their biggest problem is, and they'll usually complain about how hard it is to find good programmers. "There's just nobody out there," they say. "I can't hire anyone."

Frankly, this is baloney. In microeconomic terms: the market is clearing. There are hundreds of thousands of programmers out there, and you can hire them, if you know how. This article is about the "programmer's perspective" on how to find people and convince them to work for you.

First of all, let me motivate you a little.

If you go down to your neighborhood Starbucks (a large coffee shop chain) and ask them if they have job openings, they'll say something, "yes, we have n openings now," where n is a nice round integer like "0" or "4". Once they hire n people, they don't have any more openings and they won't hire any more people. That's because for a given coffee shop, customer demand is fixed. You have to have enough employees to meet that demand, or the lines will be too long and people will walk out, but after that point, hiring more people is pointless.

But with software companies, that's not the case. Customer demand is directly related to how many problems you solve, how good your product is, and how many features it has that address potential customers' needs. Adding another programmer means you have time to implement more features, optimize and debug code, make file format converters, and other things which get you more customers. So almost every software company or Internet startup I've visited has generally had a policy of "we'll hire any good technical people we can get." (I wrote an article about this which you can read here.)

Now, let's review some microeconomics. In a free market, it is almost axiomatic that the market always clears. That's a technical term that means that when somebody tries to sell something, if they are willing to accept the market price, they will be able to sell it, and when somebody wants to buy something, if they are willing to pay the market price, they will be able to buy it. It's just a matter of both sides accepting the market price.

The trouble comes when people are not realistic about market prices. A two bedroom, two bathroom coop apartment in my neighborhood sells for about $500,000. Sellers who believe that will be able to sell their apartment. Sellers who don't believe that and list their apartment for $800,000 are going to wait years without selling their apartment, and they won't know why.

A top notch web programmer in New York City at a top web development firm, working on a consulting basis, is going to bill about $250 an hour. People who believe that are going to be able to hire consulting firms to build their web sites. People who don't believe that aren't going to understand why they can't hire anyone.

You can hire programmers if you are willing to pay at or above the market price. This past spring, a company in Texas took out full page ads in the Yale Daily News and the Harvard Crimson saying "Attention seniors: We are going to hire the best 5 computer science graduates, and we're going to give them a $200,000 per year salary and a BMW." This happens to be a lot more than the going rate for recent CompSci graduates of those schools (which is more like $65,000), but I'm sure they filled those positions.

"Well, yes," you say, "if I'm willing to vastly overpay, I can hire people. But how do I hire people without shelling out $200,000 a year?"

Aha! We're onto something here. The trick is that money is not everyone's number one motivator, in fact, it's not even the number one consideration for most people. There are other things that matter more, and, luckily, most of those things are cheaper than money. So the next trick is, how do we whittle down that market price to something a little bit more affordable by substituting the kind of non-monetary benefits that excite people?

There are a lot of other things that you can "give" programmers, for example, you can give them the feeling that they're working on something worthwhile, which has actual cash value for most people and therefore reduces the amount you have to pay them.

I love Julia Roberts. And I loved her the most when she threw a tantrum in the movie Erin Brockovich:

"That is my work! my sweat! my time away from my kids! If that's not personal, I don't know what is!"

And this is the key to making people want to work for you: you have to understand that people spend most of their waking hours at work, and they are going to take it personally, and they are not going to be willing to suffer throughout their career just for a little extra cash they can spend when they retire. Work needs to be rewarding and pleasant. Doing things that make work rewarding and pleasant is the most important part of attracting people.

I recommend a three-pronged approach to hiring people:

  1. Make the workplace attractive,
  2. Eliminate obstacles, and
  3. Provide benefits which are more valuable than the money they cost.

1: Make the workplace attractive

After I finished

It reminded me of a visit I made to EDS once when I was working for Microsoft. EDS had modern, clean, well lit offices, but they were just seas of cubicles. Any kind of personalization of the workspace was forbidden. Fluorescent lights everywhere. Windows were strictly for managers. If you've seen the movie Office Space, you know what I mean. As I left the building with my Microsoft colleagues, I remember saying, "you know, if I had to work in a place like that, I'd cry for two hours when I got to work in the morning."

There's a whole list of things you can do to create a pleasant and attractive workplace. People spend so much of their time at work that you can't blame them for wanting to work in a nice place, not a

Cubicles have become such an icon of nasty workplaces that it's shocking that the companies who manufacture them still have the chutzpah to pretend that they're efficient, productive, and pleasant. Peopleware calls this "lying by repeated assertion."

Put yourself in the job candidate's shoes. Company number 1 shows you a big crowded room, with a bunch of desks shoved in tightly, lots of marketing guys shouting on the phone next to the programmers and a bunch of sales jocks shouting tasteless jokes. "You would sit here, in this cubicle with Doris from Accounts Payable." Doris has a whiny, high-pitched voice and self-affirmation notes to herself pinned up on the dividers. The half-height cubicle they show you is completely lit by a fluorescent light which is on the fritz.

Company number 2 shows you through a quiet hallways, shows you a sunlit, plush office with a window, big plants, a door that closes, a nice Aeron chair, the smell of fresh-brewed coffee from the espresso machine in the kitchen, cool mahogany stuff and Ansel Adams prints, and says "this would be your office." There's a patio outside for lunch when it's warm and a lounge where people meet for afternoon tea. Doris is nowhere to be seen.

All else being equal, which job are you going to take?

It's human nature for people to jump to snap judgments based on all kinds of external factors. If they come to visit you on the interview and something just doesn't smell right, they're not going to work for you. First impressions count for a lot. The fountain in the lobby, the fully-stocked kitchen with Snapple and Perrier as well as cheap drinks like Coke -- it all seems trivial, but it's making an impression.

2: Eliminate Obstacles

Make a list of the obstacles that are likely to keep someone from wanting to work for you, and figure out creative ways to solve them.

First of all, get your recruiting department to stop being an obstacle. I don't know how many companies I've seen that have disorganized recruiters who just forget candidates and don't call them back. It took my last employer something like 3 months from the time I first contacted them to make me an offer. There's just no excuse for that.

Another thing I don't understand is why people have so much trouble telling candidates "no thanks." You would not believe how many companies I know that just don't tell the candidate anything, hoping they "get the hint." They do this even when there's a devoted recruiting staff whose whole job is saying "no thanks" 93 times a day.

When a candidate applies for a job, you should be able to schedule an interview immediately. After the interview, you should be able to give the candidate a straight yes or no answer on the spot. If you have to wait for references, you should be able to take care of that in a day or two, and keep the candidate apprised daily.

Next, find out what things are worrying your job candidates. In the US, a common problem is getting work visas for people from other countries. Get a great immigration lawyer and pay all expenses.

In some markets, especially Silicon Valley and New York City, the housing situation is insane. Apartments are absurdly expensive and real estate brokers are sharks. This worries people. Solve the problem for them: offer them temporary, furnished housing for a couple of months while they look for an apartment. Retain an local real estate expert to find them housing -- somebody who you trust will only show them nice apartments, not ratty dumps, and pay all the broker fees.

What other fears do they have? A new school for the kids? A job for their spouse? All these are things you can help with by being creative. You can hire relocation specialists who are experts at smoothing these things over (but make sure they're good, or you'll do more damage than good).

Want to hire someone who has been working in Europe? They're probably used to 6 weeks a year of paid vacations. Match it. Giving somebody six weeks instead of three weeks vacation is like paying them about 6% more salary. Many people who like to travel or spend long weeks at the beach with their family would be happy to get more vacation in exchange for lower base salary, and they'll be more productive to boot.

3. Provide benefits which are more valuable than the money they cost

Have you ever gotten junk mail from a phone company offering you a $50 savings bond if you switch to their service?

Most people think that a $50 savings bond is worth $50. Nope. It's worth $25.

Have you ever gotten a free T-shirt at a trade show?

Most people think that a t-shirt is worth about $15, because that's how much they cost at the Gap. Nope. It's worth about 50 cents.

There are a whole bunch of things you can give employees which cost less than they seem to be worth. Almost every software company has free soft drinks. Massages seem to be popular in Silicon Valley. I've heard of a consulting company that is setting up beach houses and ski houses for employees to use. Microsoft has subsidized espresso carts in practically every building. Google has a gourmet chef with free lunches and dinners. Health club memberships cost a lot less than you think if you buy them for all your employees, because only a fraction of them actually take advantage of them.

Every Microsoft manager has a morale budget to use as they see fit. Whenever a new Star Trek movie comes out, Microsoft rents out an entire movie theater for the afternoon and takes the whole damn company out to the movies. Movie theatres are usually empty in the afternoon, so this costs a heck of a lot less than you think.

Think about what you can do to show people how much you appreciate them. Here's an example. When you call someone up to say, "you've got the job," if you're organized, you probably have a little post-in note stuck on your finger you got from The Big Boss which says something like $100-$105. That's to remind you that you're going to offer them $100,000, but if they balk, you have the authority to go as far as $105. Now imagine that you call the candidate up and offer them $100. They accept. Great! You saved the company $5,000 a year!

Oh my. Big deal. $5,000 a year. Wait a day, call them right back, and say, "I know you accepted at $100, but we've decided to set your salary at $105."

Aha. Now sit back and watch the law of reciprocity kick in. Having agreed that the "fair" price for their services is $100, you've thrown them a gift of an extra $5, and they're going to feel some level of extra obligation towards you in exchange. It's human nature. And it's going to keep them from taking another job. (If you haven't heard about the law of reciprocity, read Cialdini.)

And then think about the things that attract employees which do not even cost money. Give people authority. Let them work with smart people and learn from the best. Have an exciting project to work on. Make sure they keep learning and are always doing something new, not just the same old thing. Read all the other essays on my site about making a healthy work environment. Read Peopleware and make everyone else read it, too.


What Does CityDesk Do?

Friday, October 12, 2001

CityDesk is a Windows program that makes it easy to manage a web site which changes often.

With CityDesk, people who don't know anything about HTML or web servers can easily add, edit, and remove articles from a web site, using a program that is as easy as a word processor. CityDesk will automatically apply standard formatting, according to templates you provide, to those articles. Then it copies them to your web server automatically. Each article can include pictures, sounds, and other media elements.

Here are just a few examples of the kinds of things you can build and maintain with CityDesk:

  • A daily newspaper, a monthly newsletter, or a web-based magazine
  • A personal journal or weblog
  • A company's list of job openings
  • A real-estate agent's list of currently available properties
  • A company worldwide knowledge base

The real power in CityDesk comes from the fact that you only design the formatting of your site once. After that, it's easy to create new articles which use the same design. Because CityDesk keeps the text of the articles separate from their design, you can change the design in one place, and every article on your site changes accordingly.

Once you set up your site, updating it is just as easy as using a simple Windows-based word processor. The built-in word processor is WYSIWYG ("What You See Is What You Get") and includes a spell checker, word counter, find and replace, and formatting commands.

CityDesk is built around a powerful, robust database engine. This means that different people can update the site simultaneously without any risk of conflict or corruption. You can even have a whole newsroom banging away at the same time; even a virtual newsroom, with contributors all over the Internet!

When you need to manage a site with several different versions, CityDesk is invaluable. Suppose you want to publish a newsletter every week in two languages. CityDesk provides a nice interface for translators where they can see a list of untranslated articles and translate them on the spot in a split-screen environment. If you try to publish the site before everything has been translated, you'll see a warning.

Or suppose you produce a newsletter that has different regional editions. Most of the articles are the same, but you don't want to run your poem "Every Civilized Person Loves the New York Yankees" in the Boston edition. CityDesk keeps track of that and publishes both editions completely automatically.

CityDesk can also keep track of articles that need to be held until a certain date. Just write the article and set the date range, and CityDesk will never publish it before or after those dates. This is a good way to keep fresh content on your web site when you're on vacation.

Many web sites appear in multiple formats. For example, you might have:

  • the normal web version
  • a "printer-friendly" web version without ads or navigation elements
  • a version for Palm Pilots without pictures
  • a large-font version for the web site for people who like larger fonts
  • a printed version that you hand out on street corners to passers-by (who use it to line bird-cages, those scurvy knaves)
  • and so on and so forth.

Setting up template families for each of these versions is a one-time operation. After that, you only have to type in the article once, and it will automatically be published in each and every format without any additional work. Ta da!

We think you'll find that CityDesk is an extremely powerful tool, but one that's quite easy to understand. There's a tutorial which will take you on a whirlwind tour in about ten minutes. After that, you'll only need to consult the documentation occasionally.

CityDesk will be in beta from Oct 15 - Nov 30, 2001, after which it will be shipping. Click here if you would like to apply to be a beta tester.


What is the Work of Dogs in this Country?

Saturday, May 05, 2001

How naive were we?

We had assumed that Bezos was just reinvesting the profits, that's why they weren't showing up on the bottom line.

Last year, about this time, the first big dotcom failures started to hit the news. Boo.com. Toysmart.com. The Get Big Fast mentality was not working. Five hundred 31-year-olds in Dockers discovered that just copying Jeff Bezos wasn't a business plan.

The past few weeks have felt oddly quiet at Fog Creek. We're finishing up CityDesk. I'd like to tell you all about CityDesk, but that will have to wait. I need to tell you about dog food.

Dog food?

Last month Sara Corbett told us about the Lost Boys, Sudanese refugees between 8 and 18 years old separated from their families and forced on a thousand mile march from Sudan, to Ethiopia, to Sudan, to Kenya. Half died on that trip, of hunger, thirst, alligators. A few of them were rescued and delivered to places like Fargo, North Dakota, in the middle of winter. "Are there lions in this bush?" one asked, riding in a car to his new home from the airport.

Peter touched my shoulder. He was holding a can of Purina dog food. ''Excuse me, Sara, but can you tell me what this is?'' Behind him, the pet food was stacked practically floor to ceiling. ''Um, that's food for our dogs,'' I answered, cringing at what that must sound like to a man who had spent the last eight years eating porridge. ''Ah, I see,'' Peter said, replacing the can on the shelf and appearing satisfied. He pushed his grocery cart a few more steps and then turned again to face me, looking quizzical. ''Tell me,'' he said, ''what is the work of dogs in this country?'' [New York Times Magazine, April 1, 2001]

Dogs. Yes, Peter. Fargo has enough food, even for dogs.

It's been a depressing year.

Oh, it started out so amusing, we all piled into B2B and B2C and P2P like a happy family getting in the Suburban for a Sunday outing to the Krispy Kreme Donut Shop. But wait, that's not even the amusing part, the amusing part was watching the worst business plans fail, as their stock went from 316 to 3/16. Take that, new economy blabbermouths! Ah, the schadenfreude. Ah, the glee, when once again, Wired Magazine proves that as soon as it puts something on the cover, that thing will be proven to be stupid and wrong within a few short months.
Ooh, sorry, did you buy the Wired Index?
And with this New Economy thing, Wired really blew it, because they should have known by then what a death kiss their cover was for any technology or company or meme, after years of touting smell-o-rama and doomed game companies and how PointCast was going to replace the web, no, wait, PointCast already replaced the web, in March 1997. But they tempted fate anyway, and didn't just put the New Economy on the cover, they devoted the whole goddamn issue to the New Economy, thus condemning the NASDAQ to plummet like a sheep learning to fly.

But joy at other's misfortune can only entertain us for so long. Now it's just getting depressing, and I know the economy is not officially in a depression, but I'm depressed, not because so many stupid startups went away, but because the zeitgeist is depressing. And now we have to eat dog food instead of Krispy Kremes.

Which is what we're doing, because life goes on. Even though everybody's walking around with their chins glued to their chests, mourning about the hours they devoted, ruining their health and love lives for the sake of stock options in SockPuppet.com, life goes on. And the product development cycle must go on, and we at Fog Creek are getting towards the part in the product development cycle where you have to eat your own dog food. So for a while we're Dog Creek Software.

Eating your own dog food is the quaint name that we in the computer industry give to the process of actually using your own product. I had forgotten how well it worked, until a month ago, I took home a build of CityDesk (thinking it was about 3 weeks from shipping) and tried to build a site with it.

Phew! There were a few bugs that literally made it impossible for me to proceed, so I had to fix those before I could even continue. All the testing we did, meticulously pulling down every menu and seeing if it worked right, didn't uncover the showstoppers that made it impossible to do what the product was intended to allow. Trying to use the product, as a customer would, found these showstoppers in a minute.

And not just those. As I worked, not even exercising the features, just quietly trying to build a simple site, I found 45 bugs on one Sunday afternoon. And I am a lazy man, I couldn't have spent more than 2 hours on this. I didn't even try anything but the most basic functionality of the product.

Monday morning, when I got in to work, I gathered the team in the kitchen. I told them about the 45 bugs. (To be fair, many of these bugs weren't actual defects but simply things that were not as convenient as they should have been). Then I suggested that everybody build at least one serious site using CityDesk to smoke out more bugs. That's what is meant by eating your own dog food.

Here's one example of the kind of things you find.

I expect that a lot of people will try to import existing web pages into CityDesk by copying and pasting HTML code. That works fine. But when I tried to import a real live page from the New York Times, I spent a whole day patiently editing the HTML, finding all the IMG links (referring to outside pictures), downloading the pictures from the web, importing those pictures into CityDesk, and adjusting the IMG links to refer to the internal pictures. It's hard to believe, but one article on that web site contains about 65 IMG links referring to 35 different pictures, some of which are 1 pixel spacers which are very difficult to download using a web browser. And CityDesk has a funny compulsion to change the name of imported pictures internally into a canonical number, and it doesn't even have a way to find out what that number is, so the long and the short of it was that it took me one full day to import a page into CityDesk.

It was getting a bit frustrating, so I went and weeded the garden for a while. (I don't know

Wow.

That's why you eat your own dog food.

And when Michael started importing some sites himself, he found about 10 bugs I had baked in by mistake. For example, we found web sites that use complicated names for pictures that cannot be converted to file names when you import them because they contain question marks, which are legal in URLs but not legal in file names.

Sometimes you download software and you just can't believe how bad it is, or how hard it is to accomplish the very simple tasks that the software tries to accomplish. Chances are, it's because the developers of the software don't use it.

I have an even more amusing example of failing to eat dog food. Guess what email product was used internally at Juno Online Services? [If you're just tuning in, I worked on the Juno client team for a few years].

Hmm, did you guess Juno? Since that was, uh, our product?

No. A couple of people, including the president, used Juno at home. The other 175 of us used Microsoft Outlook.

And for good reasons! The Juno client was just not such a great email client; for two years the only thing we worked on was better ways to show ads. A lot of us thought that if we had to use the product, we would have to make it better, if only to stop our own pain. The president was very insistent that we show popup ads at six different points in time, until he got home and got six popup ads, and said, "You know what? Maybe just two popups."

AOL was signing up members at a furious rate, in part because it provided a better user experience than Juno, and we didn't understand that, because we didn't eat our own dog food. And we didn't eat our own dog food, because it was disgusting, and management was sufficiently dysfunctional that we simply were not allowed to fix it, and at least make it tolerable to eat.

Anyway. CityDesk is starting to look a


Where do These People Get Their (Unoriginal) Ideas?

Wednesday, April 19, 2000

Maybe I should call this article "why I should stop reading Upside Magazine". I really have been trying to stop, but they send it to me for free, and I really needed bathroom reading material, so I grabbed it and found one of the most wrong-headed articles I've seen in a long time. Well, actually, there are wrong-headed articles all over Upside magazine, but this one was particularly irksome.

It's an article by Stephen James called "Lessons in Survival". (March 2000 Upside). Every month, I'm told, Mr James will share with us "some of the simple lessons [he's] learned from [his] own startups".

It includes lots of extremely useful advise that you would never figure out on your own, like, don't spend a lot of money on office space and try to find a neighborhood where you don't have to stand in line at the local restaurants. If you are starting a dot com, Mr James reminds us, "Get out of the house... Working at home sucks." And he points out that you should never pay more than $1.50 per square foot. Thanks, Mr James! It's extremely strange to me that you could expect the same price in every market. Maybe he just assumes that everyone in the world starts all companies in Silicon Valley.

"Forget the free coffee and drinks. Sure they're free at Microsoft... who wants to be like Microsoft?"

Hello? Is this a joke? Did Upside schedule their April issue for May?

I guess Mr James lives in a funny la-la world where there are millions of programmers just dying to work at your startup. When you are the founder of a company, you want to skimp on frills; they seem like a waste of money to you. That's fine. But don't think that candidates interviewing at your company will have the same emotional attachment; they won't. They are looking for a nice place to work. Skimping on free soft drinks, a completely standard benefit at most high tech companies, is a great way to send your employees and potential employees the message that you just don't care about being an attractive workplace.

Everybody in Silicon Valley seems to be talking about Charlie, the gourmet cook at Google who used to work for Jerry Garcia. I can tell you, the food is spectacular even by Michelin standards, not just corporate cafeteria standards. Because the food in the company cafeteria is so good, people don't leave work to eat. They eat with their colleagues, which increases learning and communication. They get back to work within half an hour, making them more productive. They feel like Google cares about them, making them more loyal.

Meanwhile, "Build-outs are a bad idea," Stephen James tells us. "Don't build out your space with walls or partitions -- leave it open... If employees need an office with a door, let them go to law school or Apple."

Guess what? They will go to Apple! And replacing just one of them will cost on the order of $50,000 in recruiting and training. A friend of mine provides gorgeous private office space for his programmers in the most expensive real estate district in the USA, Manhattan, and it's probably costing about $6,000 per person per year. Not much in the scheme of things.

What's worse, Mr James seems to be completely ignorant of the documented productivity gains provided by giving knowledge workers space, quiet, and privacy. The classic software management book Peopleware documents these productivity benefits extensively. (To be fair, Mr James is not alone in his cluelessness. He's just reflecting the folk wisdom prevalent in the Valley).

Here's the trouble. We all know that knowledge workers work best by getting into "flow", also known as being "in the zone", where they are fully concentrated on their work and fully tuned out of their environment. They lose track of time and produce great stuff through absolute concentration. This is when they get all of their productive work done. Writers, programmers, scientists, and even basketball players will tell you about being in the zone.

The trouble is, getting into "the zone" is not easy. When you try to measure it, it looks like it takes an average of 15 minutes to start working at maximum productivity. Sometimes, if you're tired or have already done a lot of creative work that day, you just can't get into the zone and you spend the rest of your work day fiddling around, reading the web, playing Tetris.

The other trouble is that it's so easy to get knocked out of the zone. Noise, phone calls, going out for lunch, having to drive 5 minutes to Starbucks for coffee, and interruptions by coworkers -- ESPECIALLY interruptions by coworkers -- all knock you out of the zone. If you take a 1 minute interruption by a coworker asking you a question, and this knocks out your concentration enough that it takes you half an hour to get productive again, your overall productivity is in serious trouble. If you're in a noisy bullpen environment like the type that caffinated dotcoms love to create, with marketing guys screaming on the phone next to programmers, your productivity will plunge as knowledge workers get interrupted time after time and never get into the zone.

With programmers, it's especially hard. Productivity depends on being able to juggle a lot of little details in short term memory all at once. Any kind of interruption can cause these details to come crashing down. When you resume work, you can't remember any of the details (like local variable names you were using, or where you were up to in implementing that search algorithm) and you have to keep looking these things up, which slows you down a lot until you get back up to speed.

Here's the simple algebra. Let's say (as the evidence seems to suggest) that if we interrupt a programmer, even for a minute, we're really blowing away 15 minutes of productivity. For this example, lets put two programmers, Jeff and Mutt, in open cubicles next to each other in a standard Dilbert veal-fattening farm. Mutt can't remember the name of the Unicode version of the strcpy function. He could look it up, which takes 30 seconds, or he could ask Jeff, which takes 15 seconds. Since he's sitting right next to Jeff, he asks Jeff. Jeff gets distracted and loses 15 minutes of productivity (to save Mutt 15 seconds).

Now let's move them into separate offices with walls and doors. Now when Mutt can't remember the name of that function, he could look it up, which still takes 30 seconds, or he could ask Jeff, which now takes 45 seconds and involves standing up (not an easy task given the average physical fitness of programmers!). So he looks it up. So now Mutt loses 30 seconds of productivity, but we save 15 minutes for Jeff.

Anyway, I fully expect that most of you, reading this, will write to say, "what the heck are you doing reading Upside anyway? You get what you deserve". How true. Serves me right.


Wordsworth Responds

Monday, August 07, 2000

In Strategy Letter I, I wrote:

If you're going into a market with no existing competition, lock-in, and network effects, you better use the Amazon model, or you're going the way of Wordsworth.com, which started two years before Amazon, and nobody's ever heard of them.

I was happy to hear back from Sanj Kharbanda over at Wordsworth, who wrote:

Thanks for noticing us!

... A customer of ours pointed [Strategy Letter I] out and after the appropriate period in which I agonized over your comment I thought I might write back.

I have no qualms with what you say, though I must add that folks have heard of us (not to the extent they have heard of Aaaamaazon, but there are folks out there who know us---largely because of our reputation in the physical world). Our website does fairly well, in spite of us. We have put very little in terms of monetary resources in it and we are every marketer's nightmare.

Are there days where we don't kick ourselves? I'd be lying if I said no. Did we envision that the web would be this big for books? We knew it would be very big, we did not think it was going to be this HUGE.

Our error, we had the vision we just did not execute. (spilt milk---sour grapes etc). The only reason I can come up with it the same reason Wordsworth books has never tried to "chain" itself...we are an independent store that tries to keep the "community store" ethic and we were afraid we would loose that.

My rather extensive reply to Sanj... more of a rant about independent booksellers in general:

I love the concept of independent bookstores, but in many cases I think that they are just not "doing what it takes" to be competitive -- online or in the stores. I suspect that is because the types of people who love books enough to be indie booksellers are not necessarily the best businesspeople.

Here's an example I can think of... a famous bookstore here on the Upper West Side, Shakespeare and Co., closed down when a Buns and Noodle's SuperDuperStore opened up practically next door.

I liked Shakespeare and Co., but, you know what? They just weren't doing what it takes to be competitive. They didn't have places to sit down. They made you check your bags. They didn't have a cafe. Their selection was much smaller than B&N -- for example, I wanted a book about bicycle touring -- B&N had a whole shelf of bicycle books; Shakespeare had one book.

Indies "claim to fame" is that they have more knowledgable staff and they do better selection of interesting books. Sometimes true, but not in the case of Shakespeare and Co., who were hiring the same local entry level workers that B&N did.

At the time, Shakespeare had limited floor space to expand their inventory -- but a giant store that would have been perfect for a large bookstore was vacant, right across the street. My guess is that Shakespeare was undercapitalized and couldn't afford to compete on the same basis as B&N. They hung on for a while but eventually closed down. I would miss them, if I could think of a single thing they offered that B&N didn't.

In the online world, the same thing seems to be happening between Wordsworth and Amazon. Doc Searls changes his bookstore from Amazon to Wordsworth and sees his revenue plummet to $0. (Jacob Nielsen explains why). I myself am an Amazon affiliate; it's earning me about $100 a month which just about covers the cost of all the books I buy at Amazon :) [By the way... you guys should be thankful for Nielsen's free UI advice; normally people pay about $30,000 for this kind of advice!]

Anyway, no matter how much I love independent bookstores, they just weren't getting the books to the people. In many smaller cities across America, the Borders and B&N megastores represent the first time there's been a decent selection of books available. I love the fact that B&N means that worried gay teenagers can read XY magazine, even if they live in Kansas City. I love the fact that B&N means that 2600 magazine is available nationwide. I love the fact that I can buy an XML reference manual 11:30 PM in my neighborhood... before B&N, even in New York City, you had to go to McGraw Hill in midtown for good computer books, and they closed at 5 PM promptly.

Anyway, that's my bookstore rant :) It's great to hear from you, and I suspect a lot more people will hear of Wordsworth if Amazon runs out of money as quickly as some analysts think they will!


Working on CityDesk, Part Five

Tuesday, November 13, 2001

As promised I've been writing a series of articles about the creation of CityDesk. The primary purpose of these articles is to share the reasoning behind some of the decisions we made as a software company, and the techniques we used in developing our flagship software product. Today I want to talk about the five guiding principles that shaped the architecture.

One of the biggest decisions we made about CityDesk was to make a standalone Windows executable. With all the hubbub about web services, not to mention the noisy Internet dotcom boom we just lived through, this was a non-obvious decision. Why is CityDesk one of those old-fashioned programs you download and run, instead of, say, a web site you go to, in the style of Blogger or Atomz? Or a web server you install, like the Big Iron content management systems (Interwoven, Vignette StoryServer, etc.)?

An even more non-obvious decision was not to use XML. People who were a little bit too close to Silicon Valley groupthink looked at us like we were crazy because CityDesk is not based on XML and is not delivered through a web browser.

Now we're thanking our lucky stars that we didn't have a bunch of stupid venture capitalists forcing us to copy all the other content management companies, and we're grateful that we're not in Silicon Valley where everyone meets at Bucks and Stanford University seminars and copies each other's bad ideas, because the one thing we've heard from everybody who's tried CityDesk, consistently, is that CityDesk is the easiest content management software they've ever seen, full stop. And we got this ease-of-use because we believed certain things about software.

1. You can't do good UIs in a web browser.

When we started building CityDesk we kept hearing one thing about Big Iron content management systems, which were all web-browser-based. Despite the fact that these systems had web-based editing interfaces and advanced workflow, we kept hearing that in real life, reporters created their content in Microsoft Word, did the workflow the old way (email), and only at the very last minute, a secretary opened the Word file and cut and pasted it into the content management system.

Why?

Because nobody wants to compose in a big TEXTAREA on an HTML page.

There are too many things that are impossible to deliver properly in a web browser. First of all, there's the latency. Even if the web server is running on your own machine, round trips still take a certain amount of time, so all web applications feel somewhat sluggish. Second, there is no decent text editing widget for web browsers. With CityDesk as a Windows program, I can give you a feature where you can drag a picture from the desktop into an article. There is no way to create that kind of user interface through the web. I can keep a word count in the corner of the screen and update it in the background whenever you stop typing. You can use Ctrl+S to save without losing your place in the document -- something many writers have learned to do regularly. (Good luck creating a word processor inside a web browser that doesn't instantly lose everything, without prompting, if the user closes the browser). CityDesk has menus. Remember menus? And they work exactly like you expected them to, because the web browser doesn't have it's own menu, with a bunch of irrelevant commands, that wants to eat the Alt key. We don't have to waste any screen real estate on browser geegaws (like "back" buttons and spinning e-globes) that have no meaning for CityDesk. We get our own icon in the taskbar instead of looking like all the other web browsers you have open. I could go on for days about the nice UI things we can do in a Windows application that just can't be done with a web browser. In fact there's a whole chapter about it in the printed edition of my book.

"Sure," you say, "you can't get as slick an app through the web, but you get ubiquity! People can go to an Internet cafe in Phuket and ..." Yeah, OK, that sounds great for Hotmail. And in the future we will build a typical compromise-laden web interface to CityDesk for those times when you're at the beach in Thailand. In the meantime, my number one priority is to make an application that's easy to use so that people like it, so that they tell their friends about it, so that everybody buys it and we have to hire full time employees just to slit open the envelopes and take out the checks.

2. XML is a Dumb Format For Storing Data

I'm not sure why XML got so sexy. It has its advantages; it's sure a good idea for data interchange or for all those little files you need to store settings. But for real work it just can't do what a solid, multiuser, relational database can do. The next time some uninformed analyst at Gartner or Giga or Forrester tells you "in the future, everything will be XML," ask them how to do "SELECT author FROM books" fast with XML. Hint: you can't. It has to be slow. XML is not the way to store a lot of data. Now tell me how to insert a new book at the beginning of the table without massive bitblts. Of course, I doubt if there is an analyst in one of those companies who would even understand that sentence, but that's life. In the meantime, CityDesk uses a relational database. Over and out.

3. Most People Don't Control Their Web Servers

Of all the people in the world who need to put data up on the web, only a tiny percentage of them have any control whatsoever over what software runs on their web server. We decided that CityDesk has to be an all-client application because probably 95% of the people who maintain web sites simply do not have the ability to install a content management system on their web server, even if they wanted to. We reduced the server requirements of CityDesk to (a) any web server that knows how to serve static files and (b) ftp or file copy access to that server. We think this will open up content management to a whole new universe of potential users.

4. If You Can Only Do One Platform, Do Windows.

I'd love to have a Mac version and a Linux version, but they are not good uses of limited resources. Every dollar I invest in CityDesk Windows will earn me 20 times as many sales as a dollar invested in a hypothetical Mac version. Even if you assume that Mac has a higher percentage of creative and home users, I'm still going to sell a heck of a lot more copies on Windows than I could on Mac. And that means that to do a Mac version, the cost had better be under 10% of the cost of a Windows version. Unfortunately, that's nowhere near true for CityDesk. We benefit from using libraries that are freely available on Windows (like the Jet multiuser ACID database engine and the DHTML edit control) for which there are no equivalents on the Macintosh. So if anything, a Mac port would cost more than the original Windows version. Until somebody does something about this fundamental economic truth, it's hard to justify Mac versions from a business perspective. (Incidentally, I have said time and time again, if Apple wants to save the Mac, they have to change this equation.)

And don't get me started about Linux. I don't know of anyone making money off of Linux desktop software, and without making money, I can't pay programmers and rent and buy computers and T1s. Despite romantic rhetoric, I really do need to pay the rent, so for now, you're going to have to rely on college kids and the occasional charitable big company for your Linux software.

5. Content Management Can't Require Programmers

One of the reasons Big Iron content management costs so much is the team of programmers it takes to figure out the thing and set it up. Even the free and low-cost content management systems are designed by geeks for geeks. People have said to me that there's no market for CityDesk because "with XML and XSL it's a solved problem."

Uh huh. Thanks. Even I can't figure out XML and XSL, and I'm pretty sharp.

We decided that everything in CityDesk has to be simple enough for web designers and HTMLers, who are not necessarily programmers. And for the end-user, even HTML is too much. To the end-user CityDesk can't be any more complicated than a WYSIWYG word processor.

The rule of thumb with ease-of-use is that if you make your program 10% easier, you'll double the potential number of users of your product. In CityDesk we made absolutely no compromises on ease of use. Sure, it's not the most powerful system out there. We'll compromise on that. And you can't update your blog from an Internet Cafe in Bali. Another compromise. But I guarantee you that any HTMLer can create CityDesk sites, and anyone who can use a word processor can manage a site that was created for them by an HTMLer, and you'll never need a programmer.

Summary

All in all we made a bunch of decisions based on a world view that, frankly, was not very popular during the Internet boom. There was a time when you couldn't get funding for a company that wasn't a web pure play. And the lemming VCs thought that they had to do what all the other VCs were doing: pure web interfaces. To which I say, Thank you! Thank you for keeping your dumb overfunded companies out of my way, because now, over here in GUI land, I can't find any real competition, and now you're not funding anyone, so it's smooth sailing for Fog Creek.

Working on CityDesk Part: 1 2 3 4 5


Working on CityDesk, Part Four

Monday, October 29, 2001

Boy, what a terrible weekend. I've come down with a cold. Fever. Runny nose. General malaise. And WININET.DLL.

WININET.DLL, you see, is a software file provided by Microsoft. It comes with Internet Explorer and with all versions of Windows since about 1996, and it was the bane of my weekend. And it illustrates a fascinating fact about how much software developers' day-to-day lives have changed in the last decade.

Here's what happened. Late on Friday afternoon I was setting up some new web servers designed to handle the increased load we're expecting when we ship CityDesk. One of the first things I did was try to use CityDesk to copy files to the new servers using the FTP protocol. We had a firewall set up that prevented FTP from going through. CityDesk froze up.

"No problem," I thought, "I'll use passive-mode FTP, which can get through that firewall."

That's when I noticed CityDesk doesn't support passive-mode FTP.

"OK, how hard can that be to implement? It's probably available as an option on the file transfer library we're using." One checkbox and I'm done.

But ... where's the checkbox?

No mention of it in the documentation.

Searching the object file itself didn't turn up anything likely.

I checked Google Groups, formerly known as DejaNews. It's a complete archive of every UseNet discussion. This is where programmers ask each other questions about the most arcane topics. As I told Babak once: it's a big world out there. You're never the first person to have this problem.

After about five minutes searching, the conclusion was inescapable. Our file transfer library, which Microsoft gives away for free with the Visual Basic compiler, can't do passive-mode FTP.

OK, back to the drawing board. What are my choices? I wrote up a list:

  1. Do without Passive FTP. This would make CityDesk useless to a large number of people, something I wasn't willing to do.
  2. Purchase a commercial FTP library. Honestly, I've always had bad luck with commercial libraries, having discovered one too many times that their code quality is rarely up to the meticulous standards we set for Fog Creek. When I looked around the discussion groups where developers were discussing other FTP libraries, they always seemed to have scary bugs that I couldn't live with.
  3. Use Microsoft's other file transfer library, the infamous WININET.DLL. This is actually what Microsoft Internet Explorer uses to transfer files, which, despite its dismal reputation, is used so widely that I thought it had to be reasonably bug free (by now, at least). Anyway, lots of programmers use WININET.DLL and if I run into trouble I'm sure to find ample discussion of the problems on DejaNews, er, Google Groups.

I thought that #3 seemed painless enough. In fact I was already using WININET.DLL somewhere else in our code to import web pages, using the HTTP protocol.

A search of Microsoft's online knowledge base revealed that you can't really do FTP with WININET.DLL from Visual Basic code; it does some complicated stuff with threads that mean that you have to call it from C or C++. I thought the easiest thing to do would be to create my own custom FTP control written in C++, which talked to WININET. I chose to use Microsoft's ATL library to create the custom control, because it makes the smallest files. ATL is the most complicated programming environment in the world, requiring a brain the size of Colorado and 10 years of solid experience to understand what's going on. I have studied ATL in depth three, maybe four times in my career and I can never remember all the bizarre template crap that's going on in there. Nobody can.

Yes, Virginia, it is possible to create a software development environment which is so difficult to use that no human being can do it. ATL and COM+ are my two favorite examples (the latter is so complicated that only one man on Earth, Don Box, actually understands everything that's going on). C++ itself comes pretty darn close. But most programmers are too macho to admit this.

Luckily, Microsoft provides some wizards with ATL which write all the hard code for you. If you want to do something unusual, you're on your own, so my motto was Nothing Unusual. No sudden moves. Just add some simple methods and events and get the hell outta there, hopefully with my brains non-exploded.

At one point, after I had written almost half the code, I discovered that one of the checkboxes that I had checked on the wizard which wrote the inscrutable ATL code for me was wrong. But once the code is written, it's written. For the life of me, I couldn't figure out what the checkbox had done and where to change it in the code. I searched MSDN (a gigabyte of documentation on programming Windows which I keep on my hard drive), the online knowledge base, and finally the entire Internet using Google, and didn't find an easy way to change it. So I created two entirely new projects using the wizard, checking the box in one case and not checking it in the other, and then ran WinDiff, which compares two entire directories listing all the differences, to find what the checkbox really changed so I could change it in my code. (Somewhere, I had to change a hardcoded number to 131473, because I wanted my controls to be visible at runtime. A classic example of why COM programming is not for humans.)

The Microsoft documentation for WinInet was pretty decent, as these things go, but not decent enough. In the page documenting FtpOpenFile, you find both of these mutually contradictory quotes:

  1. "No file handle is returned"
  2. "Return value: Returns a handle if successful"

Well, which is it? Empirically, it wasn't returning a file handle.

The next thing I discovered is that if there is a packet filter (a simple form of firewall) somewhere between you and the server, the code hangs trying to copy files. That's normal; it's the way of the Internet; there's nothing you can do about it. After a minute, WinInet will realize that packets are not getting through and will time out. But the user is likely to get impatient long before a minute is up and hit the Cancel button.

When you hit the Cancel button, my code tells WinInet to give up and close down the connection. But as I discovered, if you do this in one of these packet-filter situations, WinInet will simply crash, bringing your program down with it. It's clearly a bug in Microsoft's code. An exhaustive search of all my Internet sources found a couple of people reporting the same crashing behavior, but nobody had a workaround.

How can that be? I thought. Since Internet Explorer uses the exact same code, wouldn't Internet Explorer crash in the same situation?

I tried it. What I discovered is that Internet Explorer doesn't crash in this situation -- it shows an hourglass and freezes for a couple of minutes, waiting for the time-out it knows it will get. This proves that Microsoft's programmers knew about this bug in WinInet and worked around it, instead of just fixing the code in the first place. Stupid stupid stupid. For the umpteenth time, I found myself dependent on a code library which had a crashing bug that was unacceptable in code I shipped. What are you supposed to do if you're the chef at Les Halles and your fishmonger is giving you smelly fish?

Another two hours of investigation and experimentation. Finally I decided that in this case, when the user hits the Cancel button, instead of freezing like Internet Explorer, I will simply hide the file transfer so it looks like the operation has been cancelled. In the background, invisible to the user, I'll wait around for the time-out to happen.

So, as I said, developers' lives have changed. All weekend I couldn't sleep. Tossing and turning in sweat-drenched sheets, I had feverish ATL nightmares. Sunday morning I got up at 3 am and coded for 4 hours just to avoid the bad dreams about Structured Exception Handling.

Ten years ago, to write code, you needed to know a programming language, and you needed to know a library of maybe 50 functions that you used regularly. And those functions worked, every time, although some of them (gets) could not be used without creating security bugs.

Today, you need to know how to work with libraries of thousands of functions, representing buggy code written by other people. You can't possibly learn them all, and the documentation is never good enough to write solid code, so you learn to use online resources like Google, DejaNews, MSDN. (I became much more productive after a coworker at Google showed me that you're better off using Google to search Microsoft's knowledge base rather than the pathetic search engine Microsoft supplies). In this new world, you're better off using common languages like Visual Basic and common libraries like WinInet, because so many other people are using them it's easier to find bug fixes and sample code on the Web. Last week, Michael added a feature to CityDesk to check the installed version of Internet Explorer. It's not hard code to write, but why bother? It only took him a few seconds to find the code, in VB, on the Web and cut and paste it.

We used to write algorithms. Now we call APIs.

Nowadays a good programmer spends a lot of time doing defensive coding, working around other people's bugs. It's not uncommon to set up an exception handler to prevent your code from crashing when your crap library crashes.

Times have changed. Welcome to a world where the programmer who knows how to tap into other people's brains and experience using the Internet has a decisive advantage.

Working on CityDesk Part: 1 2 3 4 5


Working on CityDesk, Part One

Friday, October 12, 2001

I've been rather quiet lately on this weblog -- mainly because we've been working so hard at Fog Creek getting ready for the beta of CityDesk, our flagship product. But I'd like to spend some time now talking about the design and development of CityDesk, because it's a great case study for the kind of software development practices that I've been advocating here for more than a year. Over the next few columns, I'll focus on "The Story of CityDesk," with a look at some of the behind-the-scenes, day-to-day stories of a real software development project.

We're launching the beta on October 15th -- just three days away! This is officially On Time. We made a schedule for shipping CityDesk way back in June. We figured out estimates for all the remaining tasks and bug fixes, added them up, and got October 15th.

Every couple of weeks, we checked through our task list, revised estimates, and so forth. We've found a lot of bugs over that time and added a lot of small features, but we've also killed a lot of features that we just won't have time for. And lo and behold, now we are almost completely done. Most of what we're doing these days is getting set up to run the beta. Michael added a dialog box to CityDesk for sending feedback. He also wrote some very spiffy code that catches any unhandled exception on any copy of CityDesk running anywhere in the world, prevents the app from crashing, and pushes the exception info up to our FogBUGZ bug tracking database here in New York City. I've been fixing bugs, writing the help file, and writing some pages for our corporate web site that will explain what CityDesk is and why you would buy it.

A common misconception, I assume popularized by Hollywood, is that as you get closer to shipping software, activity becomes frenetic as everybody scrambles to finish all the things that need to be done in time for the deadline. In the typical crappy movie, there's a mad rush of typing in a room full of cool alterna-dressed programmers with found-object earrings and jeans jackets. Somebody stands up and shouts to the room in general "I need the Jiff subroutine! Somebody give me the Jiff subroutine!" A good looking young woman in Vivienne Tam urbanwear throws a floppy disk at him. "Thanks!"

As the second hand swoops towards the :00, the whole team waits breathlessly around Ryan Phillipe's computer and watches the "copy" progress indicator as the final bits are put onto a floppy disk with less than a second to spare before the VC cuts off funding.

I suppose some software shops have last-minute coding frenzies like this. If so, their software is probably marked by incredibly poor quality. No code works the way you expected it to work the first time, and if you're writing code up until the last minute, it's going to suck. The first Netscape open source release, documented in the excellent movie Code Rush , demonstrates this.

On good teams, the days before shipping just get quieter and quieter as programmers literally run out of things to do one at a time. (Yesterday I took the day off to explore New York City with my wee niece and nephews.)

The number of new bugs being found has decreased to a point at which we feel confident about releasing the beta. It is crucial to get to zero known bugs (what Netscape famously called "Zarro Boogs") before releasing a beta. If you don't, you'll waste a lot of time during the beta reading 200 emails about a bug that you already knew about. And you've just used up time and goodwill of those 200 beta testers, so they may not bother telling you about the next bug they find, something you didn't know about. Or the bug may stop them from trying other parts of the program that needs some pounding. This seems self-evident, but almost every time I've been on a real product, everybody starts to think that releasing the beta on time is more important than releasing a Zero Known Bugs beta. (After all, it's ok to have bugs in the beta, they say. And I agree: it is ok to have bugs in the beta, just not known bugs.)

I'll keep posting The Story of CityDesk over the next few days, keep an eye out for frequent updates.

Working on CityDesk Part: 1 2 3 4 5


Working on CityDesk, Part Three

Wednesday, October 17, 2001

Like all good hackers, I have been programming since junior high, using my dad's account on the University of New Mexico IBM 360 mainframe. But the first real computer science course I took was during the first year of college. The course covered C and assembler. It had hundreds of students, most of whom, like me, had been fooling around in Pascal or Basic since they were toddlers.

The course proceeded merrily until one day, the professor introduced pointers.

Dread.

Suddenly, the majority of the students in the class were in deep trouble. For some reason, some people just do not seem to be able to write code with pointers in it. They were born without the part of the brain that does indirection, I guess.

Since then, I've mentally divided the world into three groups. The largest group of people can't program at all. There's another, smaller group of people who can program, but not with pointers. And there's a tiny group of people who can program, even with pointers. Those elite few can even understand what it means to write CString*& in C++.

My first job at Microsoft was putting a decent scripting language into Excel. Although I was given free rein to implement whatever language I saw fit, we went with Basic for several reasons. (1) We had a Basic compiler team in house. (2) You didn't need pointers, which meant that (3) more people are comfortable using Basic than any other language. Indeed Visual Basic is the best-selling language product of all time.

Visual Basic is an extremely productive way to write code, especially GUI code. Want bold text on a dialog box? It's one click in VB. Now try doing it in MFC. You have to create a subclassed control, it's a big mess, you have to know all about LOGFONTS and Windows window subclassing and a bunch of other things and you need about three lines of code once you have the magic class.

But many VB programs are spaghetti, either because they're done as quick and dirty one-offs, or because they're written by hack programmers without training in object oriented programming, or even structured programming.

What I wondered was, what happens if you take top-notch C++ programmers who dream in pointers, and let them code in VB. What I discovered at Fog Creek was that they become super-efficient coding machines. The code looks pretty good, it's object-oriented and robust, but you don't waste time using tools that are at a level lower than you need. I've spent years writing code for C++/MFC and years writing code in Visual Basic, and let me tell you, VB is just much, much more productive. Michael and I had a good laugh today when we discovered somebody selling a beta crash-reporting product at $5000 for three months that Michael implemented in CityDesk in two days. (And we actually implemented a good part of ours in C++/ATL). And I also guarantee you that our Visual Basic code in CityDesk looks a lot better than most of the code you find written in macho languages like C++, because we're good programmers, and we write comments, and our variable names are well-chosen, and we do things the simple way, not the clever way, and so forth.

I'll go out on a limb here. In my years of experience, I have seen many language and programming fads come and go. But there's only ONE, that's right, ONE language feature I've ever seen that actually improves your productivity significantly. No, it's not object oriented programming; no, it's not intentional programming or assertions or programming by example or CASE or UML or XML or Java. The only thing that improves your programming productivity is using managed code - that is, using a language in which memory management is automatic. Java and .NET languages do this with garbage collection; VB does this with reference counting; I don't care how you do it, just let me concatenate strings without thinking about where the new bigger string will go and I'll be happy.

One of the things about Visual Basic is that it doesn't always give you access to the full repertoire of Windows goodies that you need to make a polished application. But what it does do, better than almost any other programming environment, is let you drop into C++ code (or call C APIs) when you're desperate or when you need that extra speed. For example, you can always get the HWND of a control and do native stuff to it, which is not the case in Java. As another example, a lot of the non-GUI, time-sensitive inner loops, like the word counter, in CityDesk are actually implemented in C or C++ for speed. This ability gave us the confidence to use Visual Basic even though it can't do everything and it tends to do string processing slowly. But since we're all C++ programmers, we have no fear of creating a DLL or OCX to count words, or parse script, or call a Windows API. So about 5% of CityDesk is actually in C or C++, and we'll probably move a little bit more of the code to C++ to speed up a few more inner loops.

Now, Visual Basic is not the perfect programming language. It's fairly object oriented but there are little things that you can't do with it, like have base classes with implementation reuse. It's limited to Windows. And the worst part about coding in VB is that people think you're not cool because your code doesn't have {'s and }'s. I can live with the shame if it means I'm more productive.

Philosophically, I think that C# has a bright future in the Windows GUI programming world. It's not as embarrassing as VB, and it uses the type of syntax which C/C++/Java programmers have come to love. VB programmers looking to upgrade can't upgrade painlessly to VB.NET, because there are so many major differences in the programming environment. Even Microsoft admits that you can't port from VB to VB.NET, you have to rewrite. And that's enough of a pain that many VB programmers will use this opportunity to look around at what else is out there. I think many will choose C#, because it's virtually the same language as VB.NET with slightly different syntax and vastly less stigma attached to it.

What about Java? Yes, I've used Java extensively, but unfortunately the language, the code libraries, and especially the GUI libraries are just too primitive for a commercial desktop application. I like the language and I appreciate the benefit of write-once-run-anywhere, but frankly not a lot of desktop software is sold for Sun Solaris and I think that WORA benefits Sun more than it benefits software developers, and I'm not willing to write an app that behaves in an inferior way on 95% of my customer's computers to benefit the 5% with alternate platforms. Every Java app LOOKS like a Java app, takes forever to launch, and just doesn't feel completely native. Since CityDesk's competitive advantage comes from having an excellent GUI, that's one area where I refuse to skimp.

I am virtually certain that I will now receive a million emails from lovers of tcl/tk, or Delphi, or C++ Builder, or NextStep, or Cocoa, or perl, or Python, or RealBASIC, or some other programming environment which may or may not be suitable for creating a professional Windows GUI. That's nice. I don't really want to get into a debate about language features or programming environment features -- at some point, you have to stop debating and write code! I'm just trying to explain why we chose to use VB for the GUI part and C++ for time sensitive bits of code. If you do want to have a fun religious war over programming languages, please do so on the discussion group!

Working on CityDesk Part: 1 2 3 4 5


Working on CityDesk, Part Two

Saturday, October 13, 2001

I've been trying to get a discussion board running for three days now, without much success. Installing server software is just much harder than installing Windows software. There are always multiple complicated steps that involve permissions, accounts, database servers, dependencies (do you have the absolute latest version of perl?), superuser, and web servers. If you're using free software, nobody wants to volunteer to make a decent setup program (or even documentation, half the time), so you're generally on your own there. And when you're using commercial software, the vendor would usually like to sell you a three-week integration consulting project so they can make another $50,000 copying some files and editing some configuration files. (We have a double-click SETUP program for FogBUGZ that works pretty well, but there are still people with funny configurations where it doesn't set all the permissions correctly.)

I'm also a bit particular about what discussion software we use. The purpose of the board will be a place for CityDesk beta testers (and later, users) to ask questions, provide feedback, and share ideas. In designing a UI for anything, the very first question you always want to ask is: who is the user? Specifically: are most users casual, occasional users, or are these users who will be spending all of their time using your program? For casual users, learnability and simplicity are more important than usability and power. In that sentence, by "learnability," I mean, the ability for novices to figure out how to get tasks done rapidly. By "usability," I mean only the ability to do tasks in a convenient and ergonomic way without making mistakes and without needing to do repetitive tasks. A data entry system that minimizes keystrokes by prefilling things and automatically jumping from field to field is more usable for experienced users, but it's harder to learn because it behaves unexpectedly to a novice.

Most people using the Fog Creek bulletin board will be going there to ask a question or raise an issue. Certainly in the early days, learnability and simplicity are our priority. When I looked a bunch of discussion boards, I found that most of them have their heritage in the BBS world, where the same people log on to chat for 4 hours every night. Those people love features like a geegaw that lets you put a graphical smiley in your posting, and the ability to upload snapshots of their ugly mugs to appear next to their postings, and the ability to click a button and never see postings by the blithering idiot who wrote this one again. All of these features are neat for power users but they just clutter up the interface for novices.

(Longtime Juno email users may have noticed that as time went on, the big

I spent Wednesday and Friday playing around, installing various buggy BBS systems, some of which required a Linux server, others Windows 2000, playing with DNS to move discuss.fogcreek.com hither and yon, figuring out why DNS caches weren't flushing, installing and reinstalling database servers, and generally getting frustrated. One of the most popular packages, Discus, actually hardcoded its own URL in so many places that it needed to be reinstalled from scratch just to change the URL. (In fact, it wasn't enough to reinstall it... you had to redownload it. The download package already had your personal URL hardcoded throughout.) And it had a perfectly terrible UI in which there was a little treeview showing folders (so far so good)... but each folder was actually a command, not a container. A broken metaphor, worse than no metaphor at all. That had to go. Then I tried IdealBB, a decidely beta package. I wouldn't ordinarily run software that is advertised as beta, but this thing seemed to have been through many releases and it was alleged to be very close to shipping. Michael and I actually had to roll up our sleeves and debug the thing ourselves to get it sort of running, but then there were too many ASP errors and it had a tendency to crash the server. (Too bad, because it is one of the finest looking discussion boards, visually). Finally I flirted with Manila, since we have it running anyway for our weblogs, and we've already written a little daemon which watches Manila and restarts it when it crashes (about once every two days). But (as far as we could tell) Manila requires membership to post a message, and in my experience that is enough to turn away 90% of the casual visitors who might otherwise use the discussion board. It would be great for small elite communities of people who all post all the time, but I don't want anything to get in the way of a beta tester casually reporting a bug.

The system I like best, believe it or not, is Lusenet, by Philip Greenspun, because it's just super simple. That's what I've been using for the Joel on Software discussion group. There are a couple of reasons we couldn't use that. One, it is really not ready for prime time. There are actually things you can do in Lusenet that still show you the Oracle statement they just executed, as if Philip left in some debug printfs. Second, it's not hosted on our own servers and we run the risk of it going away, taking our valuable data with it. Third, to host it on our own servers requires AOLServer and Oracle, and we don't have the former, and can't afford the latter.

When I got home, grumpy from all the time I had wasted, I realized that the software I want consists of exactly one table, and I could write the thing myself in less time than in took me to install some of these packages. Which I did. Two hours of work (ASP, Microsoft Access, and VBScript) and I had banged out a system that did pretty much everything I wanted (which is not much!) Check it out at http://discuss.fogcreek.com/joelonsoftware. (If you have trouble reaching it that's because of all my DNS messing around. You'll have to wait a couple of days for caches to flush.)

There's a lesson in here somewhere, but I'm already well past 1000 words. In the past I wouldn't have cared about word counts because I didn't know what they were, but now I'm using CityDesk which keeps a running word count in the corner of the screen. So we'll talk about the lesson tomorrow!

Working on CityDesk Part: 1 2 3 4 5


Worst Project Ever?

Wednesday, September 25, 2002

Hi Joel,

I'm sure that this is perplexing. As Napster's vice president of engineering and CTO during much of this time, please let me share an alternative perspective. Building a paid music site based on peer-to-peer technology involved more work than you might think at first glance. We had to:

  1. Integrate a user management and billing system.
  2. Build new Windows and Mac client applications that handle the new system, plus add significant improvements to usability and reliability, including better search tools.
  3. Create a system that identifies every MP3 file our users share with 100% accuracy, regardless of the names given to the files. Determine with 100% reliability if any of these files is on a black list of disallowed files, provided by recording companies and music publishers in many formats and with varying degrees of accuracy. This is not even theoretically possible, but major strides were made with the help of Relatable's acoustic fingerprinting technology and a large amount of "expert system" analysis code developed in-house.
  4. Build a payment system to compensate the various rights-holders of the media content that understands fully the vagaries of the music royalty systems and the concomitant dispute resolution processes they involve.
  5. Build a system that would enable owners of music to automatically submit both content and ownership and royalty rates to the system. Digitally encode, fingerprint, and catalog all submitted content. Seed the system with the licensed content.
  6. Build a Digital Rights Management (DRM) system from the ground up that allows controlled usage and secure distribution of content over a peer-to-peer network, including content that was introduced by users.
  7. Build a web-based user registration system.

All of this was accomplished and ready to go. I think you would be very surprised at the tiny size of the "whole team of developers" that was involved. Versions of the system were in beta test from December 2001. The only holdup to full deployment was the record companies' and publishers' refusal to license their content to Napster.

Please keep in mind that simultaneous to much of this work being done, significant development of the existing Napster system was required to be made under court order. Most of the engineering management and many of the key Napster developers were involved in this work. I was also directly involved in the legal process, requiring me to spend massive amounts of time doing legal work, including briefs, declarations, depositions, and very frequent meetings with the RIAA's and the Court's Technical Experts.

As an aside, I hope you don't think that the "3 months" Shawn took to write the original version (not true anyway) was the reliable system that scaled to handle 2 million simultaneous users! Although the Windows client was largely Shawn's, the server code was completely rewritten.

Having managed many large-scale engineering projects, I am well aware of the second-version syndrome. But in this case, we were really building a new first version.

Best,

Eddie Kessler